Graphics Question

If you want to create a look-up table in Python, use a dictionary. Dictionaries are one of the fundamental data structures in Python and are ideal for this type of thing.
 
C
Well, I'm warming up to sketch as it's a thin veneer and not very different from simply having a bunch of tried and true routines in a library. If you look at wiring though, it's pages and pages of code for a simple demo. I think python is a great deal higher level.

Regards
cww
 
C
Well, it depends on how fast the math is in python. The thing is getting pretty big already importing pygame, pyserial, math, time, etc.
And I'm still struggling with the whole OO thing. And I managed to get the Freeduino in some type of a loop where it ignores the serial port. Might be bricked, but it's only $5 or so to replace the chip. Kinda stalls the flow. I was looking at using the internal pull ups as the source resistor for the thermistors which would be convenient, but must be verbotten.

Regards
cww
 
C

curt wuollet

Well, The Chart Recorder Project is winding down. I replaced the ATmega328 chip in the Freeduino and that came back to life. It works so well that I am going to forget the game port version, although I did eventually get the sound/gameport card and it works. I will resurrect that if anyone is interested in really low cost. The four individual graph version is undergoing testing with the requester. I rehashed the program to make a 900 X 700 pixel version that fits on the screen of my ancient T23 Thinkpad. It plots all four with different "ink" on the same "graph paper". That way it would be trivial to expand to all 6 analog inputs. The thinkpad was running Fedora 6, so the new purpose provided a reason to upgrade, which was an ordeal. Well. the upgrade went without a hitch, even the wireless works out of the box. But, all the upgrade options wanted to remake the partitions so, I spent a day backing everything up first. I had a lot of stuff on the TP. I have a question for the python fans: Since Python is interpreted, is it faster to code inline or make say, 4 function calls? There are a lot of statements I repeat 4 times that cry out to be a function, but I've left them that way for clarity. I still don't have a place to put the files up, so if anybody wants the code and bitmaps before I find one, you can help me work out what needs to be done to get it running on a "virgin" Linux install. It was easy on the new Fedora 13 install on the Thinkpad, but pyserial needed hacks. I set the Freeduino up with ordinary phone connectors and boxes so the packaging and cabling come from any hardware or big box store. Total outlay for 6 channels of analog added to the PC of your choice should be less than $30 if you shop well.

Regards
cww
 
Not making a function call is always faster than making a function call. Whether or not it's worth the cost depends on how much work is being done.

If you want an Python program to run fast, you don't write it like a C program. A lot of things that you would do with a loop in C can be done with a single instruction in Python. You look at processing lists using list comprehensions or map (list comprehensions are usually faster). You look at what is in the built-in functions or the standard library. Etc. Of course, better algorithms usually trump all other methods.

The key of course is to benchmark it and see if you can find any measurable difference. There's a profiler in the standard library, but for a lot of simple algorithms you can just put it in a loop and measure it with time.time() statements.

If you want to post that part of the program here, I can have a look at it. Optimization is a good subject and no doubt other people would be interested in that aspect of the discussion.
 
C

curt wuollet

Sure I can post it.
The program runs as is on my desktop. To run on my old T23 laptop it wants a readline before the 900 iteration loop to prime things. It returns nothing and times out, but after that everything works. Go figure. I will send the bitmaps for it oob.

****************************************************************
<pre>
import pygame, sys,os,math,time,serial
from pygame.locals import *

pygame.init()

pygame.display.set_mode((900,700))
# Initialize joysticks
#class Stick:


# def _init_(self):
# pygame.joystick.init()
# self.joystickAxes = {}
# self.joystickHats = {}

# Thermistor Coefficients
C1 = .001030
C2 = .000239
C3 = .000000157

pygame.display.set_caption('Chart Recorder')
screen = pygame.display.get_surface()
bg = pygame.image.load("4in1b.png")
background = bg.convert()
cleanp = pygame.image.load("4in1.png")
paper1 = pygame.Surface((900,700),0,screen)
paper2 = pygame.Surface((900,700),0,screen)
pp = [paper1,paper2]
pp[0] = pygame.Surface.copy(cleanp).convert()
pp[1] = pygame.Surface.copy(cleanp).convert()
scale = pygame.image.load("4in1scale.png")
# Display some text
font = pygame.font.Font(None, 16)

screen.blit(background,(0,0))
screen.blit(paper1, (0,0))
pygame.display.flip()
oz1 = 350
oz2 = 350
oz3 = 350
oz4 = 350
# Setup event queque for timer and quit
pygame.event.set_allowed(None)
pygame.event.set_allowed([14,QUIT])
pygame.event.clear()
# Adjust this time, 96,000 = 24hrs per screen, 4000 = 1hr/screen, Seconds/.9
# Will set with a button etc., eventually.
pygame.time.set_timer(14,500)
step = 5.000/1024
ser = serial.Serial(port=0,baudrate=9600, timeout=20)
# Needed to "prime" laptop. No idea why.
# ser.readline()
while 1:
pp[1] = pygame.Surface.copy(cleanp)
ttext = font.render(time.strftime("%c",time.localtime()), 1, (10, 10, 10))
textpos = ttext.get_rect()
textpos.centerx = pp[1].get_rect().centerx
textpos.centery = 10
pp[1].blit(ttext, textpos)

btext = font.render(time.strftime("%H:%M:%S",time.localtime()), 1, (10, 10, 10))
textpos = btext.get_rect()
textpos.centerx = 25
textpos.centery = 40
pp[1].blit(btext, textpos)
scalepos = scale.get_rect()
scalepos.centerx = 8
scalepos.centery = 349
pp[1].blit(scale,scalepos)
scalepos.centerx = 468
scalepos.centery = 349
pp[1].blit(scale,scalepos)
for x in range(900):
pygame.event.wait()
print ser.write('A')
line1 = ser.readline()
line2 = ser.readline()
line3 = ser.readline()
line4 = ser.readline()
v1 = int(line1) * step
v2 = int(line2) * step
v3 = int(line3) * step
v4 = int(line4) * step
print v1
print v2
print v3
print v4
r1 = v1/((5.00 - v1)/10000)
r2 = v2/((5.00 - v2)/10000)
r3 = v3/((5.00 - v3)/10000)
r4 = v4/((5.00 - v4)/10000)
print r1
print r2
print r3
print r4
# last term is a calibration offset for the thermistor spread

T1 = ((math.pow((C1 + C2 * math.log(r1) + C3 * math.pow(math.log(r1),3)), - 1) - 273.15) * 1.8) + 32 -3
T2 = ((math.pow((C1 + C2 * math.log(r2) + C3 * math.pow(math.log(r2),3)), - 1) - 273.15) * 1.8) + 32 -3
T3 = ((math.pow((C1 + C2 * math.log(r3) + C3 * math.pow(math.log(r3),3)), - 1) - 273.15) * 1.8) + 32
T4 = ((math.pow((C1 + C2 * math.log(r4) + C3 * math.pow(math.log(r4),3)), - 1) - 273.15) * 1.8) + 32 + .18
print T1
print T2
print T3
print T4
screen.blit(background,(0,0))
screen.blit(pp[0],(-x,0))
screen.blit(pp[1],(900-x,0))
z1 = int(-((T1 - 70.0) * 4.0) + 270)
z2 = int(-((T2 - 70.0) * 4.0) + 270)
z3 = int(-((T3 - 70.0) * 4.0) + 270)
z4 = int(-((T4 - 70.0) * 4.0) + 270)
pygame.draw.line(pp[1],(255,0,0),(x -1,oz1),(x,z1),1)
pygame.draw.line(pp[1],(0,255,0),(x -1,oz2),(x,z2),1)
pygame.draw.line(pp[1],(0,0,255),(x -1,oz3),(x,z3),1)
pygame.draw.line(pp[1],(0,0,0),(x -1,oz4),(x,z4),1)
pygame.draw.circle(screen,(255,0,0),(900,int(z1)),5,0)
pygame.draw.circle(screen,(255,0,0),(900,int(z2)),5,0)
pygame.draw.circle(screen,(255,0,0),(900,int(z3)),5,0)
pygame.draw.circle(screen,(255,0,0),(900,int(z4)),5,0)
oz1 = z1
oz2 = z2
oz3 = z3
oz4 = z4
pygame.display.flip()
#if event.type == QUIT:
# sys.exit(0)
ct = time.strftime("%H%M%S",time.localtime())
pygame.image.save(screen,ct)
pp.reverse()
</pre>
*****************************************************
Efficiency isn't too big a concern with an event interval of 96 seconds, but at .5 seconds it uses some CPU. Even with the inline coding, it's still remarkably compact for what it does. Could get some more speed by converting all bitmaps, but weird things happen.

Regards
cww
 
I will show the program re-written several different ways. I'll just do the calculations, as I don't have the hardware to run the entire program with live readings. I have added in a dummy serial data class to simulate the readings. I don't know what the data format is (I assume it's just an integer represented as an ASCII string).

First, here's the common section, with various constants and the dummy serial port class.

<pre>
#################################################

#!/usr/bin/python

import math, time, operator

class serial():
"""This is a dummy class to simulate the serial port.
"""
def __init__(self):
pass
def readline(self):
return '350'
# This substitutes for the serial port.
ser = serial()

# Thermistor Coefficients
C1 = .001030
C2 = .000239
C3 = .000000157

step = 5.000/1024

# This is the number of iterations to run the test.
iterations = 100000

###############################################
</pre>

Now, I'll show the original code with the graphics calls left out.

<pre>
##############################################

# In-line instructions.
starttime = time.clock()


for x in range(iterations):

line1 = ser.readline()
line2 = ser.readline()
line3 = ser.readline()
line4 = ser.readline()

v1 = int(line1) * step
v2 = int(line2) * step
v3 = int(line3) * step
v4 = int(line4) * step


r1 = v1/((5.00 - v1)/10000)
r2 = v2/((5.00 - v2)/10000)
r3 = v3/((5.00 - v3)/10000)
r4 = v4/((5.00 - v4)/10000)

# last term is a calibration offset for the thermistor spread

T1 = ((math.pow((C1 + C2 * math.log(r1) + C3 * math.pow(math.log(r1),3)), - 1) - 273.15) * 1.8) + 32 -3
T2 = ((math.pow((C1 + C2 * math.log(r2) + C3 * math.pow(math.log(r2),3)), - 1) - 273.15) * 1.8) + 32 -3
T3 = ((math.pow((C1 + C2 * math.log(r3) + C3 * math.pow(math.log(r3),3)), - 1) - 273.15) * 1.8) + 32
T4 = ((math.pow((C1 + C2 * math.log(r4) + C3 * math.pow(math.log(r4),3)), - 1) - 273.15) * 1.8) + 32 + .18


z1 = int(-((T1 - 70.0) * 4.0) + 270)
z2 = int(-((T2 - 70.0) * 4.0) + 270)
z3 = int(-((T3 - 70.0) * 4.0) + 270)
z4 = int(-((T4 - 70.0) * 4.0) + 270)


oz1 = z1
oz2 = z2
oz3 = z3
oz4 = z4


print 'Elapsed time for in-line is ', time.clock() - starttime

print z1, z2, z3, z4

</pre>

###########################################

Now here's a version using map. Map is a built in function that operates on an entire "list" of data. It takes one or more lists as inputs, applies the specified logic to each one, and outputs a list as a result. A list is like an array (except the list elements don't have to be all of the same type).

You will notice the use of the "lambda" key word. That is a way of specifying a simple anonymous function. That is, it's like creating a function in place that doesn't have a name. For example:

vdata = map(lambda x: int(x) * step, linedata)

This takes the data in the "linedata" list (which has 4 elements), and feeds it one at a time to "lambda x: int(x) * step". This is equivalent to doing something like this:

<pre>
def calcv(x, step):
return int(x) * step
</pre>

You will also notice that I imported the "operator" module. This provides math operators as functions. That lets me do this:

<pre>
Tdata = map(operator.add, T1data, caloffsets)
</pre>

That takes 2 lists, passes one pair of numbers at a time to "add", which then adds them together.

Lambda can handle multiple lists at once as well (e.g. "lambda x,y: x+y"), so I could have combined this with the previous operation. However, I wanted to make this more directly comparable with the next example.


##########################################

<pre>

# Using map.
starttime = time.clock()

# last term is a calibration offset for the thermistor spread
caloffsets = [32 -3, 32 -3, 32, 32 + .18]
numreadings = len(caloffsets)

for x in range(iterations):

linedata = map(lambda x: ser.readline(), range(numreadings))

vdata = map(lambda x: int(x) * step, linedata)
rdata = map(lambda x: x/((5.00 - x)/10000.0), vdata)

T1data = map(lambda x: ((math.pow((C1 + C2 * math.log(x) + C3 * math.pow(math.log(x),3)), - 1) - 273.15) * 1.8), rdata)
Tdata = map(operator.add, T1data, caloffsets)

zdata = map(lambda x: int(-((x - 70.0) * 4.0) + 270), Tdata)

prevdata = zdata


print 'Elapsed time for mapped version is ', time.clock() - starttime

print zdata

</pre>

############################################

Here's a version using list comprehensions. A list comprehension is like map, but uses a different syntax. The syntax is based on mathematical set notation. It is more flexible than map in some ways, but it won't handle two lists without a lot of faffing about, so I used map for one step.

############################################

<pre>

# Using list comprehensions.
starttime = time.clock()
# last term is a calibration offset for the thermistor spread
caloffsets = [32 -3, 32 -3, 32, 32 + .18]
numreadings = len(caloffsets)

for x in range(iterations):

linedata = [ser.readline() for x in range(numreadings)]

vdata = [int(x) * step for x in linedata]
rdata = [x/((5.00 - x)/10000.0) for x in vdata]

T1data = [((math.pow((C1 + C2 * math.log(x) + C3 * math.pow(math.log(x),3)), - 1) - 273.15) * 1.8) for x in rdata]
Tdata = map(operator.add, T1data, caloffsets)

zdata = [int(-((x - 70.0) * 4.0) + 270) for x in Tdata]

prevdata = zdata


print 'Elapsed time for list comprehension version is ', time.clock() - starttime

print zdata
</pre>

##############################################

Here's a version using list comprehensions, but with the equations altered a bit to avoid redundant operations.

##############################################

<pre>

# Using list comprehensions and optimising calculations.
starttime = time.clock()
# last term is a calibration offset for the thermistor spread
caloffsets = [32 -3, 32 -3, 32, 32 + .18]
numreadings = len(caloffsets)

for x in range(iterations):

linedata = [ser.readline() for x in range(numreadings)]

vdata = [float(x) * step for x in linedata]
rdata = [x/((5.00 - x)/10000.0) for x in vdata]

rdatalog = map(math.log, rdata)
T1data = [((math.pow((C1 + C2 * x + C3 * math.pow(x, 3.0)), - 1.0) - 273.15) * 1.8) for x in rdatalog]
Tdata = map(operator.add, T1data, caloffsets)

zdata = [int(-((x - 70.0) * 4.0) + 270) for x in Tdata]

prevdata = zdata


print 'Elapsed time for optimised list comprehension version is ', time.clock() - starttime

print zdata

</pre>

############################################

The alternate methods are a lot briefer, but what about speed? I ran these in a loop (see the "iterations" variable near the beginning) and measured the time for each using time.clock(). Here are some typical results (the numbers will vary between consecutive runs).

In-line: 3.63 seconds.
map: 4.3 seconds.
list comprehensions: 3.77 seconds.
optimised: 3.21 seconds.

Map was 18% slower.
List comprehension was 4% slower
The optimised version was 12% faster.

For the optimised version I did 2 things. One was in the first equation it converted the value to a float instead of an int. I'm not sure if that is correct for the application, but it avoids unnecessary type conversions later. I also changed several constants from integers to floats for the same reason.

I also split up the main equation so that the "log" function is calculated separately. This avoids calculating it twice.

List comprehensions are usually faster than map, especially if you are using lambda with map. A lambda is like making a function call (it has to set up a stack frame), so there is extra overhead. However, it isn't that much slower here because so much of the real work is in the math equations themselves.

When you use map with a built-in function or with a library like "operator" however, then map can be very fast. So, there's no simple rule of thumb for this.

The final results are in a list called "zdata" (I tried to keep the names close to your originals). If you want to access individual elements of zdata, you can index it as zdata[0], zdata[1], zdata[2], etc.

The end result of this is you are expressing each equation only once, and it is probably much more compact than using function calls (because is processes all 4 calls in one line).

If you want to take more readings, the calculations automatically accommodate them.

I have put the serial port read operation in a map or list comprehension. That looks a bit "odd" but it will probably work. If you want 4 separate calls you can do this and still have the data in a list:

<pre>
# Create an empty list.
linedata = []
# Add the readings.
linedata.append(ser.readline())
linedata.append(ser.readline())
linedata.append(ser.readline())
linedata.append(ser.readline())
</pre>

You can also do this:

<pre>
linedata = [0, 0, 0, 0]
linedata[0] = ser.readline()
linedata[1] = ser.readline()
linedata[2] = ser.readline()
linedata[3] = ser.readline()
</pre>

 
C
Hi Michael

I looked at these features with great interest, because I believe you code best when you capture the spirit of the language you are using. That's why I prefer the old pre ansi C (K&R) style and I'm trying to catch the spirit of Python. These constructs do look a little bit forced for this program, but would be more useful for a larger, more complex dataset. I did do the float types and math deconstruction and generally cleaned things up. The employment situation had me tense the other night and I couldn't sleep so I threw myself into a more challenging project. I now have a virtual circular chart recorder which stretched my memory of trigonometry almost to the edge. After a couple of approaches that were foiled by the lossy rotate transform, it works pretty good, needing only to copy over the serial code to be functional. Mechanical Cir. chart recorders are an anachronistic hack that has had surprising staying power mostly for limit auditing and the like. It too is surprisingly small and I can post it if there is interest. Coding and problem solving till the sun comes up do let you sleep the sleep of the exhausted. It's amazing what you can do with a page of Python, but it was a slow page and will need comments for anyone to see that there is method to the madness. Issues with blitting square pegs onto round holes and the like :^)

Regards
cww
 
I had a look at that again, and the "math.pow(x, 3.0)" should have been "math.pow(x, 3)" like in the original. I should have been paying more attention when adding decimal points. Also, the thermistor offsets should have been real numbers as well in order to avoid a type conversion.

If I was going to do this however, I would probably calculate a look-up table on start up and put it in a dictionary (hash table). That would be about as fast as you can get.

If you have an application that needs to do a lot of number crunching (this example is pretty simple by comparison), then the normal solution is to use "numpy" or "scipy" (the two are closely related). That's a third party FOSS Python library that has everything - vector math, FFTs, digital filters, matrix math, etc. It will also take complex equations and compile them to run in the library outside of Python. The libraries themselves are written in C and Fortran.

Numpy/Scipy are a wrapper around the same standard scientific and engineering math libraries used by almost everyone. Some of the very expensive proprietary math packages use these exact same open source libraries as the foundation for all their calculations.

For this application, you don't need Numpy/Scipy. If you were writing a production test system that had to do this sort of math however, then that would be the solution. I would definitely pick Python over something like Labview in that type of application, because the language flexibility and overall library support are so much better.
 
C
One of the things about the modular nature of Python is that there are numerous dependencies and the transforms at least, may already use numpy. I seem to recall it being loaded as such when I installed python with yum. Besides the friend that wanted the thing in the first place, I have had use for such a thing in winter when I'm heating with the corn stove and wood stove and kerosene heaters to balance temps and monitor unheated spaces to keep pipes from freezing, etc. I have an old Compaq Armada M700 notebook that I got free at a company sponsored hazardous waste free disposal day. Someone was going to pitch it in with all the tvs and old monitors and I diverted it for just such an occasion. It runs Fedora 13 well so now all my mobile computers are up to date. It can be a semi-permanent temp recorder. So another question came to mind. I wonder if I can get someone who has Python set up on Windows to test if the program will run there as well. Not an imperative, but interesting to know. Since the Arduino stuff works on Windows, if the Python part does as well, I made something that works cross platform with zero effort. The only catch is that to really test it, you would need an Arduino as the serial part is one of the bigger questions. It serves my purposes now, but if it would be of use to others, I might be able to spend the effort to make it release grade SW with selectable times, scales, etc. While I wait for the funding to proceed with the PLC project, I could put it up on the Sourceforge site before I lose interest and move on.

Regards
cww
 
If you're using Fedora, then I think that Python should have come already installed as part of the base system. I'm not sure how they do their packages (I'm more familiar with debs), but if it pulled in numpy then they have some additional package that pulls in extras that they thought you might want. If you installed something with yum, then it wasn't the base Python install because yum is a Python program (so it already had to be there for yum to run). I won't swear that Pygame doesn't use numpy, but I don't know what for (unless it uses it for some sort of complex game physics calculations).

All of the stuff that I posted was from the base language or standard libraries. That is 100% cross platform. Pyserial is cross platform, and I have used it to write and test software on Linux and then deploy it without any changes on MS Windows in the factory. The only "problem" that I had is that serial communications on MS Windows (using the standard Microsoft drivers) has lower performance than serial communications on Linux. That's not a practical problem however unless you need to push the serial port to the absolute limit. I don't know about Pygame, but I suspect that won't be a problem either.

Python is very good at being cross platform. Anything platform specific is usually very clearly marked as such. Python even handles things like using the correct line endings when it writes text files. You have to watch out for things like the Windows file system is not case sensitive, but that's not something that is happening outside the language itself. You're probably pretty safe publishing the program as is and saying it will probably work on MS Windows.

To be completely sure however, you have to test a complete system (including an Arduino). The biggest problems that I've had with MS Windows is that the serial and network I/O systems are just not as robust as the ones in Linux. I once had to re-design a program from the ground up because the networking system in MS Windows couldn't handle the load. I didn't find that out until final testing when I could put the complete system under full load for an extended period of time.

I've also seen MS Windows (Server 2008 and Windows 7) simply fall over and die under extended periods of heavy load. First it will kill off the application, and then the Windows subsystems themselves would one by one go to 100% CPU usage and then crash themselves until nothing was left except a blank screen (grey, not blue however). That's not something you will find however except by actual testing using your specific conditions, and those are "normal" Windows problems (not something specific to your application).

Despite the above "war stories", I don't think you'll have a problem in your application.
 
Top