From time to time we have discussions on this list concerning how PCs running general purpose operating systems respond under various load conditions. The most recent was conducted under the subject "PC: Labview+servo control". The usual concern expressed is that the operating system will cause the program timing to become erratic if the system is under any significant load. The program timing is a function of the operating system scheduler, and so is limited by the operating system capabilities.

Usually, these discussions proceed from annecdotal observations. I thought it would be useful to have a set of actual measurements made under controlled conditions. The systems I have available for conducting tests have a Linux OS, so these conclusions apply to Linux only. The same testing method could be applied to a computer with MS-Windows though.

My conclusions are that (for the conditions that I tested under):

1) Although the operating system used (Linux) is not a real time OS, it was under most conditions able to provide timing which is repeatable within a millisecond 99.9% of the time. This is true down to a 1 millisecond (1kHz) rate.

2) CPU load, disk activity, and network activity have no measureable effect on the ability of the operating system to schedule the test program at the requested interval. The results for tests where the PC is heavily loaded (to 100%) with any of these are virtually the same as for the no load condition.

3) Keyboard and mouse activity do appear to have some measureable effect. with delays of up to 10 milliseconds in one half of one percent of samples. These effects are probably not significant for most applications however.

These results are unexpected and counter-intuitive. I would have expected *no* effect from the keyboard or mouse, and at least some degrading of performance when under heavy CPU, network or disk load. The overall system under all conditions is also much more repeatable than I would have predicted.

Any comments on the above would be appreciated.

If anyone is interested in repeating these tests under different conditions, I can provide the test programs that I used. The actual measurement program is a small 'C' program I wrote for this test that I can provide source for. The "load" programs are very small "bash" scripts I wrote that can be used as is, (or easily re-written if necessary).

I don't have a computer with MS-Windows that is available for running these tests. A set of test results showing the equivalent figures for MS-Windows would doubtless be of interest to a great many people doing applications on that platform. If someone does have a suitable system available and would like to volunteer to repeat these tests, let me know and we can discuss how to conduct the tests.

What follows below is a brief summary of how the tests were conducted, and the results. If anyone is interested in more details, please let me know and I will elaborate. I still have the full set of test data.

=== Method =The basic experiment consisted of running a small program which would read the PC clock, record the current time, sleep for a specified period of time, and then wake up and repeat the process. This would be the "measurement" process. Various additional "load" processes were also run at the same time to simulate computing loads which may cause scheduing deviations.

The tests were 1) "no load" - the computer is idle, 2) "CPU" - the CPU is loaded to 100%, 3) "network" - large files are continously copied from another computer to the ethernet port, 4) "disk" - data was repeatly written to disk, 5) "mouse" - the mouse was continously moved manually, 6) "keyboard" - the keyboard was typed into at a rate of 1 to 2 keystrokes per second.

Each test was repeated at sampling (scheduling) intervals of 1, 5, 10, 25, and 50 milliseconds. The mouse and keyboard tests were exceptions to this, as the loads were being applied manually, and I didn't have the patience to sit through a full test sequence. For these two I tested only at 1 and 5 milliseconds.

The measurement program would take 60,000 samples each time, write the results to disk and then exit. This resulted in a total of 1.44 million readings. Reproducing these in their entirely here would result in a rather large e-mail, so I will just summarise the results below.

=== Test Results Summary =The test results may be summarised as follows. For the "no load", "CPU", "network", and "disk" tests, 99.86 to 99.99% fell exactly on the nominal interval. 99.88% to 99.99% of samples fell within +/- 1 millisecond of the nominal intervals. The worst case results were several delays of 3 msec.

For the mouse and keyboard tests, there was a bit more dispersion. For the mouse tests 97.86% fell on the nominal, while 99.48% fell within +/- 1 millisecond of the nominal intervals. The worst case results were a single delay of 9 msec. All except three measurements fell with 4 msec.

For the keyboard tests, 99.41% fell on the nominal, while 99.62% fell within +/- 1 millisecond of the nominal intervals. The worst case results were a 3 samples which were delayed by 10 msec.

=== Test Equipment =Computer: x86-64 PC with AMD Athlon 64 3200+, 1GB RAM. Operation System: 2.6.12-12mdk, 64bit version.
Disk: 80GB SATA.

 

fascinating info! did you try any other distos? how about the RTOS patches?

i have long been interested in the possibility of using linux in automation. if i had a better grasp of the hardware needed (for i/o) i might have actually done more than just think about it.

i don't know if is actually going to happen, but there was talk amongst the engineers here of replacing some of our automation racks with newer hardware and using linux for the os.

that said, i could possibly run some benchmarks on my machines at home. i have access to a few different versions of linux, also win xp home and media center (sorry, no pro just yet). the hardware i have in mind is similar, but a little older and slower. as another possibility, i have a hard drive for my laptop (much better system) that i have been meaning to upgrade, just haven't gotten around to it. could do fresh installs and play around with it. besides, maybe i can finally get off my butt and learn how to compile a custom kernel successfully!

is there a website or something where you have more info, and/or the scripts you used? i don't know all that much about programming, but i might be able to fumble my way thru it.

dave

 

Are all the scheduler options and preemption available in the 64 bit kernel? It sounds like this scheduler prioritized user response, I believe other choices might reverse that. And I _would_ be intrigued by a comparison to recent Windows, I doubt the older versions would even be within an order of magnitude. I haven't done any such rigorous testing but using the new 2.6 kernels with some options turned on, it's better than I need, that is, I don't recall any timing errors, so I've pretty much stopped worrying about it. I should set up my original LPLC and see what the worst case cycle time is. When it gets this fast, it requires great care in timing the timing.

Regards

cww

 

Hello,

Just for completeness, did this use the Linux soft-RT features (such as sched_setscheduler(2) to set priority and mlockall(2) to disable paging) or
did it just run as an ordinary process?

Jiri
--
Jiri Baum <[email protected]> http://www.baum.com.au/~jiri

 

In reply to "Dave" - I have tried just one distro so far. I didn't think these factors would be distro specific (the basic scheduler would be common to all). I will repeat the keyboard and mouse tests with a newer kernel soon however, to see if they have changed.

I didn't try the RTOS patches either, as I was trying to see what could be done *without* using an RTOS. I think that the tests so far show that the "stock" version of Linux is more than good enough for most automation applications.

The tests were inspired by the regular discussions we have on how MS-Windows behaves erratically under various loads. These discussions are always based on annecdotal observations, so I thought a set of simple tests would be useful to put some more solid facts behind the discussion. I don't have a computer with MS-Windows which I can use to conduct the tests, so I was hoping someone else could repeat these tests for us.

I will put the test scripts and 'C' source below. They're not very big. I will include extra description of each one just above it. Hopefully the e-mail system will not scramble up the formatting too much.

I've included detailed instructions on how to use the scripts in case anyone else wants to try this and wants the detail.

This first one is the program I used to collect the timing measurements. It was written in 'c' to make sure that as few side effects as possible were introduced by the process of measurement.

Cut it from the e-mail, paste it into a text editor (use kwrite or gedit, or whatever you happen to like) and save it to disk. To compile it, open a terminal window (konsole, gterm, etc.) and type "cc loadmeas.c" (assuming you named the file "loadmeas.c". This will compile to a file called "a.out". You can use one of the many IDEs available to compile it, but this is such a short and simple program that it is probably easier to just do a command line compile. A 'C' compiler comes with almost every Linux distribution.

Rename "a.out" to "loadmeas" (with no ".out" on the end of it - the test scripts will be looking for the correct name later). To rename it, type "mv a.out loadmeas".

It should compile with no errors. I've shortened some of the comment lines below in case the e-mail adds some extra line breaks. If you get any errors
and want to e-mail back with questions on them, try to include a few lines from around the e-mail with your reply so I can see what is going on.

When you run the program, it requires a parameter specifying the measurement interval in milliseconds. So for example, "loadmeas 1" would cause the program to read the time, sleep for 1 millisecond, and then repeat (until all samples are complete). You can specify anything up to 999 milliseconds.

The number of samples is hard coded in the constant "LOOPITER". I've set this to 60,000. The program run time is the number of samples times the sampling rate, so at the maximum interval (999 ms), the test would take over 16 hours. This is why I didn't test at an interval of more than 50 milliseconds (which takes approximately one hour). If this is too time consuming, you can cut the
number of samples down to something smaller (just edit the LOOPITER value and re-compile).

When the program is done, it writes the results out to a CSV file called "tlogdat.csv". You can open it using a spreadsheet. You should note that some spreadsheets won't handle that many rows. I used Gnumeric, which had no problem with this. Again, you can cut the number of samples down if this is a problem for you.

You can try out the program by running it manually. Just type "./loadmeas x", where 'x' is the time in milliseconds. For the actual test runs however, you can use the test coordination scripts (see below) which will line up a series of tests and run them automatically.

The "loadmeas" program extends for about 178 lines, and the last line if it is "} // End of program."


// Program: loadmeas.c
// Date: 5-Oct-2006
// Version: 1.0
// Author: M. Griffin
// Description: Run in a loop for a fixed number of iterations,
// recording the time and then sleeping.The purpose
// is to assist in determing the effects of various
// loads on proces scheduling. The actual test loads
// are provided by other programs.

/*--------------------------------------------------------------------------------------
*/

#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>

/*--------------------------------------------------------------------------------------
*/

/* The number of loop iterations should be fixed at a value that
will not exceed the number of rows in a spreadsheet so that
the data can be imported into a spreadsheet for analysis. */

#define LOOPITER 60000 // Number of loop iterations.

#define DEFAULTDELAY 10 // Default time delay (in milliseconds).

#define TESTDATAFILE "tlogdat.csv" // Test results will be saved to disk.

/* Array to store test results. */
struct timeval test_results [LOOPITER];
/*--------------------------------------------------------------------------------------
*/

// Evaluate the command line arguments.
// The user is to enter one integer value to represent the
// desired delay time in milliseconds.
long eval_args(int argc, char **argv){
long delayarg;

/* The user is to supply the delay time in microseconds
as a command line arguement. */
if(argc > 2) { // Check for too many arguements.
printf("Invalid number of arguments.\n");
printf("Usage is: loadmeas <delay time in milliseconds>\n");
exit(EXIT_FAILURE );
}

if(argc < 2) { // Check for no arguements. If so, use default time delay.
delayarg = DEFAULTDELAY;
printf("No delay time specified. Using default time of %d milliseconds.\n",
delayarg);
}
else {
delayarg = strtol (argv[1], 0, 10);
}

if(delayarg < 0) { // Check for negative time.
printf("Invalid arguement. Delay time must be positive.\n");
exit(EXIT_FAILURE );
}

if(delayarg > 999) { // Check for too large of a time.
printf("Invalid arguement. Delay time must be less than 1000.\n");
exit(EXIT_FAILURE );
}

return (delayarg);
} // End of eval_args

/*--------------------------------------------------------------------------------------
*/

// Record the test results in a file for later evaluation.
// File format is ASCII CSV with the following fields:
// Sample - Data sample number, starting with 0.
// sec - System time (seconds).
// usec - System time (microseconds).
// diff (usec) - Time difference between samples in microseconds.
// diff (msec) - Time difference between samples in milliseconds.
// The first sample of diff (usec) and diff (msec) are initialised to zero.
int record_results(void) {
FILE *resultsfile; // File for recording test results.
long i, timediff;
double timeround;

printf("Load effects measurements complete. Writing test results.\n");

/* Open the file to record the test results to a file for analysis. */
if ( (resultsfile = fopen(TESTDATAFILE, "a")) == NULL) {
printf("Could not open results file.");
return(1);
}

/* Write out the data. The first record must be handled differently to
initialise
the difference calculation. */
fprintf(resultsfile, "Sample, sec, usec, diff (usec), diff (msec)\n");
fprintf(resultsfile, "0,%d,%d,0,0\n", test_results[0].tv_sec,
test_results[0].tv_usec);
for (i=1; i<LOOPITER; i++) {
timediff = ((test_results.tv_sec - test_results[i-1].tv_sec) * 1000000) +
(test_results.tv_usec - test_results[i-1].tv_usec);
timeround = timediff / 1000.0;
fprintf(resultsfile, "%d,%d,%d,%d,%.0f\n", i, test_results.tv_sec,
test_results.tv_usec,
timediff, timeround);
}
fclose(resultsfile);
printf("Test results recorded in file %s\n\n", TESTDATAFILE);
return (0);

} // End of record_results

/*--------------------------------------------------------------------------------------
*/
void estimate_testtime(long delaytime) {
long total_time;

total_time = (delaytime * LOOPITER) / 1000000;
if(total_time < 180) {
printf("Estimated test time is %d to %d seconds.\n", total_time,
total_time+1);
}
else {
printf("Estimated test time is %d to %d minutes.\n", (total_time/60),
(total_time/60) + 1);
}

} // End of estimate_testtime


/*--------------------------------------------------------------------------------------
*/

int main (int argc, char **argv){

long i, delayarg;

struct timeval current_time, // Used to store current time.
timeremaining, // Required by nanosleep.
delaytime, // Requested time delay.
delaytmp; // Temporary, as select modifies the original.

struct timezone tzone; // Needed for gettimeofday, but not otherwise used.

/*--------------------------------------------------------------------------------------
*/

// Get the command line paramter for delay time.
delayarg = eval_args(argc, argv);
// Initialise the default time delay variables.
delaytime.tv_sec = 0; // Delay time - seconds.
delaytime.tv_usec = delayarg * 1000; // Delay time - microseconds.

// Report the estimated total test time.
estimate_testtime(delaytime.tv_usec);
/*--------------------------------------------------------------------------------------
*/

printf("Starting load effects measurement test with time delay of %d
microseconds.\n", delaytime.tv_usec);

// This is where the load effects measurement takes place.
// This section runs in a loop, reading and storing in memory
// the current time.
// After each time reading, the program will "sleep" for a period
// of time (specified as a command line parameters), after which it
// will repeat the process for a fixed number of times.

sleep(5); // Sleep for a short period to allow any intial
// start-up disturbances to settle.

// Loop while recording time stamps.
for (i=0; i<LOOPITER; i++) {
// Current time of day in microsecond resolution.
gettimeofday(&current_time, &tzone);
// Store both seconds and microseconds.
test_results.tv_sec = current_time.tv_sec;
test_results.tv_usec = current_time.tv_usec;
// Sleep for requested time.
//nanosleep (&delaytime, &timeremaining);
delaytmp = delaytime; // Must re-initialise this after
// every call to select.
select(0,0,0,0, &delaytmp); // nanosleep does not work everywhere.
}
/*--------------------------------------------------------------------------------------
*/

// The test results are recorded to disk.
record_results(); // Record the test results in a file.
/*--------------------------------------------------------------------------------------
*/
} // End of program.


******************************************

The next program is a "bash" script that I called "load-cpu.bash". The last line in this one is "exit 0". Again, paste this into an editor and save it to disk. Change the file permissions to make it executable. If you are using Konqueror, right click on the file and select "is executable" under "permissions". Nautilus will have somethings similar, or you can use "chmod".

This script simply runs in a tight loop doing "x=x+1". This results in 100% CPU load. I also did some tests using a 'c' version but the results were the same, so I used the bash version in the full set of tests.

The SIGHUP signal is used by the test coordination scripts to automatically terminate the load script at the end of the series of tests. This same method is used in the other test scripts as well.


#!/bin/bash
# Load down CPU.
# 5-Oct-2006
# M. Griffin

# Terminate by sending a SIGHUP

function on_hangup () {
echo 'Hangup (SIGHUP) signal recieved, terminating.'
exit 0
}

trap on_hangup SIGHUP

X=0
while true ; do
X=`expr $X + 1`
done

exit 0


******************************************

The next program is a "bash" script that I called "load-net.bash". The last line in this one is "exit 0". Remember to make this executable (see above).

This one requires that you have the test PC networked to something else. I used an NFS mount to another PC. Pretty well every distro has some menu driven means of setting this up.

This script simply repeatedly copies the same file over and over from the remote PC to "/dev/null" on the test PC. "/dev/null" is the "bit bucket" or
black hole into which you can dump stuff. Copying to "/dev/null/" instead of to disk ensures that the load is network rather than disk I/O.

Use a very large file to ensure the load is mainly network and not CPU load (from script overhead). I used a 1GB video file, but any very large file
should do. You can use "top" or KSysGuard, or Gnome System Monitor to check the CPU usage. In the tests I did, the copying processes (NFS and bash) took about 2% or less CPU time.

You will need to edit this script to change the line "cat /home/path-to-network-mount/some-big-file > /dev/null" so that
"/home/path-to-network-mount/some-big-file" points to wherever the source file is you are copying from.

The test coordination scripts will terminate the copying automatically. It may take a few seconds before it stops, while it finishes the file copy it was working on.


#!/bin/bash
# Load down network.
# 5-Oct-2006
# M. Griffin
#
# Copy a file from another computer and dump it into /dev/null
# Copy one large file rather than several smaller ones to ensure the
# load is primarily from the network and not from executing the script.

# Terminate by seding a SIGHUP
function on_hangup () {
echo 'Hangup (SIGHUP) signal recieved, terminating.'
exit 0
}

trap on_hangup SIGHUP

while true ; do
cat /home/path-to-network-mount/some-big-file > /dev/null

done

exit 0

******************************************

The next program is a "bash" script that I called "load-disk.bash". The last line in this one is "exit 0". Remember to make this executable (see above).

This program was intended to simulate the sort of activity that occurs during simple data logging to a text file. The test could be done using a database, but that would add a lot of extraneous factors to the test.

This one writes a large block of text to disk. I've only included a small section of the text here, but you should be able to re-construct this (using copy and paste if necessary). I had 2000 lines to the text, which resulted in a text file of about 130kb. The same file is over-written each time, so there is no danger of filling up your disk as long as you have at least *some*
space.

The reason for writing a large block of text was to try to make as much of the load disk I/O with as little CPU load as possible. None the less, this still causes a lot of CPU load, at least on my PC, which has a SATA hard drive.

I tried a 'C' version of the program, and found that it still caused high CPU load, so it appears that disk I/O (at least of the type being tested) is inherently CPU intensive.


#!/bin/bash
# Load down network.
# 5-Oct-2006
# M. Griffin
#
# Create a file on disk and write to it repeatedly.
# This roughly simulates logging data to disk.

# Terminate by seding a SIGHUP
function on_hangup () {
echo 'Hangup (SIGHUP) signal recieved, terminating.'
exit 0
}

trap on_hangup SIGHUP
dummydata="
2006-10-05, 0, serial-number, data1, data2, data3, data4, data5;
2006-10-05, 1, serial-number, data1, data2, data3, data4, data5;
2006-10-05, 2, serial-number, data1, data2, data3, data4, data5;
2006-10-05, 3, serial-number, data1, data2, data3, data4, data5;
2006-10-05, 4, serial-number,

... lots more data here ...

2006-10-05, 1998, serial-number, data1, data2, data3, data4, data5;
2006-10-05, 1999, serial-number, data1, data2, data3, data4, data5;
2006-10-05, 2000, serial-number, data1, data2, data3, data4, data5;
"
while true ; do
echo $dummydata > dummydata.txt
done

exit 0

******************************************

The next program is a "bash" script that I called "cpu-load-test.bash". The last line in this one is "play ringtone.wav" (this is repeated 3 times).
Remember to make this executable (see above).

This one is a *typical* example of the test coordination scripts I used. I've just listed one of these scripts here because they are all so similar that you can just make minor edits to copies of this one to get the rest. Of course you don't need to use them at all if you want to run the tests manually.

The test coordination scripts consist of the following:

"./load-cpu.bash &".

This line starts the "load" script. You need to change this to the script name that is appropriate for the particular test. The "./" in front of the script name indicates to the OS that the script is found in the current directory. Without this, it won't find it (that is if you have a standard
configuration). This is a security feature that has been standard for most modern unix line operating systems for quite a while.

The "&" causes the script to run in the background. This allows the load program to be started up while the coordination script continues on.

The next bit is repeated a number of times.

sleep 5
./loadmeas 1
mv tlogdat.csv tlogdat-CPU001.csv
sync
play KDE_Notify.wav

First we "sleep" for a few seconds to allow the system to avoid any start-up effects. The next line starts up the "loadmeas" program with a parameter indicating the sampling interval in milliseconds (I used 1, 5, 10, 25, and 50). If you named the "loadmeas" program something else, you need to change this to the appropriate name.

The next line "mv tlogdat.csv tlogdat-CPU001.csv" renames the data logging file to something else so that subsequent tests don't overwrite it. Be sure
that you have a unique file name for each test so you don't lose your data.

The next line "sync" flushes any buffered data out to disk. This is to try to ensure the that delayed disk I/O (from the data logging) does not occur in
the middle of the next test (i.e. it tries to reduce the number of variables in the testing process).

The last line "play KDE_Notify.wav" is optional. This just plays a sound to keep me informed of the test progress. If you use this you'll need to either copy a ".wav" file into the current directory, or include the path in the line.

The last bit stops the load script after the set of tests is done:

kill -SIGHUP %+

It sends the "SIGHUP" signal ("hang up") to the last started background process ("%+"). The load scripts were written to listen for this. The network load may take a few seconds to stop if it was in the middle of copying a large file.

The "play ringtone.wav" lines are optional; again I just used them to alert me when the tests were complete.

To run the script, open a terminal window and type "./cpu-load-test.bash" (or whatever the script name is). For the no-load, CPU, network, and disk tests, the test will run completely automatically. Each of these test series takes about an hour and a half (to do all of 1, 5, 10, 25, and 50 msec).

I wrote and ran a separate coordination script for each of these (4 scripts in total) which were all slight variations on the one listed below. The "no
load" test simply left out starting up and shutting down the load creating program. You could simply write one large script to run all the tests in
series. Total run time would be about 6 hours. If you used fewer samples, it
will take less time. If you leave out the 50 msec test, it will cut the time
roughly in half.


#!/bin/bash
# CPU Load test sequence.
# 5-Oct-2006
# M. Griffin
# The purpose of this script is to create a simple computer load test
# sequence to test behaviour of the scheduler under load.

# Test 2 - Test the system under heavy load.
# The CPU load is provided by a separate script.

# Test with x milli-second period.
# Rename data log file after test
# is complete to preserve it.

#Start a background process that will load down the CPU.
./load-cpu.bash &

sync

echo "Starting load test sequence."

sleep 5
./loadmeas 1
mv tlogdat.csv tlogdat-CPU001.csv
sync
play KDE_Notify.wav

sleep 5
./loadmeas 5
mv tlogdat.csv tlogdat-CPU005.csv
sync
play KDE_Notify.wav

sleep 5
./loadmeas 10
mv tlogdat.csv tlogdat-CPU010.csv
sync
play KDE_Notify.wav

sleep 5
./loadmeas 25
mv tlogdat.csv tlogdat-CPU025.csv
sync
play KDE_Notify.wav

sleep 5
./loadmeas 50
mv tlogdat.csv tlogdat-CPU050.csv
sync

# Stop the CPU load script.
echo 'Halting CPU load.'
kill -SIGHUP %+

echo 'Test sequence complete.'
play ringtone.wav
play ringtone.wav
play ringtone.wav


******************************************


The mouse and keyboard tests were done a bit differently because I didn't have
a good way to conduct them automatically. For these I used a modified version
of the "no load" script and only tested at sample rates of 1 and 5 msec.

For the "mouse load", I just moved the mouse in a figure 8 pattern for the
duration of the test series (about 6 minutes total). For the keyboard test, I
just pressed keys on the keyboard ("q" to "m" on a "querty" keyboard) at a
normal typing rate onto a bare desktop (i.e. not into an application). Having
the script play a sound at the end of the test series is very helpful in
knowing when to stop.

******************************************

The data is saved in ASCII CSV files. Each line is one record. An example is:
"1,1160100813,844927,968,1", where the first column "1" is the sample number
(0 to 59,999), the second column "1160100813" is the current time is seconds
(from 1st of Januay, 1970), The third column is time in microseconds
(844927), the fourth column is the difference between the current time and
the previous sample (the first record defaults to zero) in microseconds, and
the last column is the same number rounded off to the nearest mill-second.

You should be able to import this into any spreadsheet that can handle the
number of rows. I imported them into Gnumeric, and used formulas to count the
number of samples (in the msec column) which fell within each range. It's a
bit difficult to explain the spreadsheet formula in an e-mail, so I'll just
say that I used a series of "=dcount" functions to count up the number of
occurances of whatever time interval (in milliseconds) I was looking for.


******************************************

It would be very interesting to see the results from MS-Windows. In fact, it
was questions about what MS-Windows would do under these conditions that
started this topic off. I and no doubt others would appreciate if you could
run at least some of the tests and report the results. Either of the XP
versions you have should be adequate for these tests.

You'll need a 'C' compiler to compile the "loadmeas.c" program. There are free
ones available for MS-Windows if you don't have one (I think that both
Borland and Microsoft give away free copies of older versions of the command
line versions).

Just about everything in the program is standard 'C'. The area that might
cause trouble for some compilers is "gettimeofday", and "nanosleep" or
"select". "nanosleep" and "select" accomplish the same thing, so you only
need to use one of them (comment out the one you are not using). Test these
to make sure they are reporting reasonable number and not just random
garbage. If nanosleep doesn't work, try using select.

The Microsoft version of "sleep" doesn't follow the POSIX standard libraries.
Basically, you need to use "Sleep" (note the capitalisation). There is an
explanation on the following web page:

http://www-128.ibm.com/developerworks/eserver/library/es-MigratingWin32toLinux.html

The Microsoft version of gettimeofday doesn't follow POSIX standards either.
Their equivalent is "_ftime".

To replace the "BASH" scripts, you could use a set of batch files. MS-Windows
batch files are not as sophisticated as BASH. but they should be able to do
this job. I don't know how to do signals with MS-Windows, but we could leave
that out with only a minor loss of convenience. It only means you would have
to start up the load scripts manually in another window, and stop them
manually (by typing control-c).

The main part of each load script could be something like:

:label
"do something"
goto label

Where, "do something" would be applying the load.

For the network test, the MS-Windows equivalent to "/dev/null" is the special
file "NUL".

So for example, the CPU test could just be:

:label

goto label

This is just any empty loop, but it should exercise the CPU quite well. Use
the MS-Windows system monitor to check the CPU load.

The network test could be:

:label
copy h:\bigtestfile.txt NUL
goto label


The disk test could be:

:label
echo $dummydata > dummydata.txt
goto label

(where $dummydata is defined similarly to the bash example above.)


If any of the above still leaves you with any questions, please let me know
and I will be happy to help further. If anyone else has something they would
like to add, please let us know.

 

In reply to Curt Wuollet - To answer your question briefly, so far as I know, all the modes available in 32 bit versions are available in 64 bit versions. However, the kernel version for the first set of test data I presented was: "2.6.12-12mdk, 64bit version". This is a little more than a year old (9-Sep-2005). I don't think much of the newer pre-emption work was merged in at that time.

To answer your questions properly, I did some more tests. These new tests were conducted on another computer with a newer kernel version 2.6.15-27-386 PREEMPT (16-Sep-2006) - less than a month old. I assume the "PREEMPT" means pre-emption patches are enabled. This is just the standard (updated) Ubuntu. This is a 32 bit version, and was run on a much slower computer (600 MHz).

This version by the way, has the 4 msec "tick" rather than 1 msec. This meant that I conducted the tests at 4, 12, and 24 msec. I omitted tests at any longer intervals than this to save time.

Repeating the tests on this platform showed the same relationships as before. That is, CPU or disk I/O loads seemed to have no measureable effect on process timing (I omitted the network I/O test in order to save time in setting up the tests). Keyboard and mouse I/O caused some small increase in outlying results, but not to a degree which should be significant for most applications (i.e. 99.97% to 99.99% of samples are within +/- 1 msec of the nominal).

The keyboard and mouse tests however showed some interesting results on an additional test. Most of the testing on this PC was conducted via an x-terminal connection (rather than connecting the mouse, keyboard and monitor directly to the PC). This was done for convenience (the previous test were all conducted with a direct connection). The keyboard and mouse tests however were repeated with a direct connection (keyboard, mouse, and monitor plugged directly into the PC). When tested through an x-terminal, the results showed a measureable increase in "spread", particularly for the mouse (for the mouse, it went from 99.99% of samples being within +/- 1 msec down to 88.62%).

One additional test was conducted on the 64 bit PC (used for the first set of tests). The PC was booted into single-user mode, and the keyboard tests conducted from the console (no GUI). The tests were repeated as before at 1 and 5 msec sampel rates. In this mode, every single sample without exception registered at exactly the nominal value (in msec). Looking at the microsecond data (which is also recorded), the total deviation is about 75 to 100 microseconds (not milliseconds). Single user mode is not very practical for normal use, but it would seem to indicate the best that can be done.

The point of these tests is not to show that "faster is better", or that a general purpose operating system can be used for genuine "real time" applications. The purpose is to show that at least some general purpose operating systems are good for most applications, and also to show what sort of loads can upset the timing by a significant degree.

The tests seem to show that for Linux at least, the timing factors that seem to make many people concerned about using PCs are not significant.

 

<p>Further reply to "Dave" - I have tried the testing with an additional distro now (Ubuntu). If you are using this in your testing, note that the current version uses a 4 msec "tick" rather than a 1 msec "tick". That means that your tests will need to be in increments of 4msec (the system will accept whatever you ask for, but it will sleep in increments of 4). You can see what the "tick" time is on your PC by just looking at the output from a test run of the "loadmeas" program. Also note that for Ubuntu, the c compiler is not installed by default in the base package. You have to use Synaptic to download it as part of the "build utils" package.

<p>See the reply on this topic that was addressed to Curt Wuollet for more information on the new test results.

<p>Also, I have some additional code that might interest you. To help when analysing the results from the additional tests, I wrote some small scripts that were easier to use than a spreadsheet.

<p>These are just a short bash script, that in turn calls a simple awk script (both are below). The awk script looks at the millisecond column in in the output file from the "loadmeas" program, and counts the number of duplicates (which gives you a count of the number of occurances for each value). The input file must be sorted before analysing it (the bash script takes care of this).

<p>The example bash script uses the "for i in" type of loop. This substitutes a series of strings into the actual command. In this example, "KEY001" and "KEY005" are substituted in place of the "$i" into "tlogdat-$i.csv" and "tlogdat-$i.txt". You'll need to edit this file to change "KEY001" and "KEY005" (and maybe "tlogdat-") to whatever is in the file names you are using.

<p>The output is a text file with the first column listing the millisecond value, and the second column listing the number of occurances of that value. It can be imported directly into a spreadsheet as a CSV file.

<p>If all this sounds confusing, you don't have to use it. It is just a quick and easy way to get summary results from the test output.

<pre>
*********************


#!/bin/bash

for i in KEY001 KEY005
do
cat tlogdat-$i.csv | sort -n -k5 -t',' | awk -f numcount.gawk > tlogdat-$i.txt
done


*********************


# Program: numcount.gawk
# Langauge: gawk
# Date: 11-Oct-2006
# Author: M. Griffin
# Description: Takes as input the sorted
# output from loadmeas.c and counts the
# number of consecutive identical millisecond
# values. Used to find the distribution of
# millisecond values for the collected data.


BEGIN {
FS = "," # Set field separator.
prev=""
cnt=0
totalrec=0
totalcnt=0
}

{
if ($5 ~ /[0-9]$/) { # Check for numeric input.
totalrec=totalrec+1

if ($5 == prev) { # Rec is same as previous.
cnt=cnt+1
} else { # New value in record.
cnt=cnt+1
if (prev!="") {print prev,",",cnt}
totalcnt=totalcnt+cnt
cnt=0
prev=$5
}
}
}

END {
if ($5 != "") { # Handle last end of file.
cnt=cnt+1
print prev,",",cnt
}

print "Total records are,",totalrec
print "Total counted are,", totalcnt
}


*********************</pre>

 

In reply to Jiri Baum - All test programs were run as normal user processes.
No soft-RT features were used. The normal (as installed by default)
background daemons were running, but all other user programs (except for the
console window) were closed before testing started.

I didn't try locking the pages in memory (mlockall). There is more than enough
RAM present that no swapping was taking place.

I used the standard scheduler for normal user processes. Using the
soft-real-time scheduler options would guaranty a more deterministic
response. However they require more knowledge to use correctly, so they are
not used for "normal" applications.

I've sent three replies back to the list (two to 'Dave', and one to Kurt
Wuollet) which haven't made it through the a-list e-mail system yet that have
more information. These include source code listings for the test programs
and the results of additional tests. One of the replies has appeared on the
control.com web site at the time of writing this, the others are presumably
in process somewhere. Some of this information may be of more help in
answering your questions (or perhaps raise more).

The test programs are very small simple ones that I wrote quickly. You may
have some suggestions for improving them.

The "load" programs were run as background processes, but so far as I know all
user programs have an equal priority (0). Using the round robin or FIFO
soft-RT scheduling would allow a user settable priority (1-99) but this was
not the objective of the test.

The objective of the tests was to demonstrate how a normal program running
with no additional fine tuning features (i.e. no use of memory locking,
standard scheduler, no changes to nice, etc.) would behave. The tests were
inspired by discussions about how various programs running under MS-Windows
seem to behave erratically as far as timing is concerned when the system is
under load.

If you have any suggestions for additional or improved tests, I would be
interested in hearing them.

 

cool! i will have to give this a go! tests underway shortly

just as a cautionary word, i am back on shift (nights, what fun!) so it will be a few days 'til i can play with the desktop. i will see what i can do with the laptop in the meantime. just did a fresh reload of it a couple weeks ago, so there shouldn't be too much garbage to fight with, and it's already set up as a dual boot with winXP (media center) and kubuntu.

i am fairly comfortable with the linux testing, having compiled a large assortment of programs on various distro's, cook-booking scrits, etc., but my experiments with trying to learn programming in a windows environment were... um, well, a little disheartening i guess. all part of what got me into linux! if i run into problems, i will try and figure it out.

i have an e-mail addy i can post here, but i can't remember it at the moment. i'll look it up when i get home.

thanks for the info, and i will be in touch.

dave

 

spent my morning getting kubuntu up to speed. i prefer suse, but they seriously screwed up the update system, and it confuses me now, when it works at all. it's all good...

my email is: spamboxx22 at yahoo dot com

i actually use a different address normally, but i don't like posting it on public forums, as i'm sure you can understand. drop me a line there, and i will email you back with the results as i get them.

dave

 

On a related note:
http://www.linuxdevices.com/news/NS9566944929.html

Regards

cww

 

Additional response time results from testing with MS-Windows.

I've done some additional testing of computer response times under load, and now have some results for MS-Windows. All the previous tests were conducted using several versions of Linux.

The following is a brief summary of the most significant observations for MS-Windows.

1) CPU load, disk activity, keyboard and mouse usage (network activity was not tested) had no obvious effect on the ability of the operating system to schedule the test program at the requested interval.

2) The operating system was usually (98.7% to 99.9% percent of the time) able to schedule the task within +/- 1 msec of the requested time.

3) While *on average* the task was run on time, each test had a number of samples with very large delays (e.g. ~ +150 msec) from nominal. That is the worst case was sometimes delayed by as much as 10 scheduling periods. There were no operating conditions which appeared to be more or less prone to this problem.

4) The nominal timer resolution for the version of MS-Windows tested was approximately 15 milliseconds. The actual resolution appeared to be somewhere between 15 and 16 msec.

My conclusions are as follows:

1) The common belief that the erratic timing experienced with MS-Windows can be attributed to specific types of loads (CPU, disk I/O, etc.) is not supported by the evidence.

2) The erratic timing (scheduler delay) for MS-Windows is random and occurs under all conditions, including when the computer is idle. There may be conditions which exacerbate the problem even more, but none were discovered in these tests. Attempts to avoid these problems by insulating the PC from the above mentioned load conditions are likely a waste of time (this doesn't mean that letting people play video games on a critical PC is a good idea however).

3) The most striking results were the occasional relatively large timing "hiccups" which occured randomly. Any application which may possibly be
sensitive to these cannot rely on brief observational "testing" to see if they are in fact a problem.

4) The limitations found in these tests appear to be due to software, and not hardware. This conclusion is based on the fact that the same tests were previously conducted on similar hardware using a Linux OS at much higher sampling rates while being generally much more repeatable. It is likely that each OS uses a different scheduling algorithm due to being targeted at
slightly different markets (i.e. entertainment and word processing for MS-Windows versus servers and embedded applications for Linux).

Although I have conducted the above and previous tests, similar tests by others would be useful to confirm these results. It would be particularly
useful to perform the network load tests for MS-Windows (I didn't do these). This should involve repeatedly copying a large file from a network drive to the NUL device. I assume the NUL device in MS-Windows will work similarly to /dev/null.

The original "loadmeas" program required modification to compile on an MS-Windows OS. Source code for the original was sent as an e-mail to the Automation List, so I won't repeat it here. For anyone who would like to repeat the tests tough, I will describe the changes I used to compile it.

*************************************

Include headers: Add #include "stdafx.h" (supposedly because its a command like program). Replace sys/time.h with sys/timeb.h. Add time.h and windows.h.

The array of test results: change the type from "timeval" to "_timeb" with the underscore).

Function eval_args: no change.

Function record_results: Since the array of test results is of a different type, the code which reads it has to change accordingly ("tv_sec" becomes "time", and "tv_usec" becomes "millitm"). High resolution time is now in milliseconds rather than microseconds, so the math which calculates the time difference has to multiply the seconds by 1000 instead of 1000000. The printing function "fprintf" doesn't like the MS time types, so I had to assign the time values to temporary variables before printing them.

Function estimate_testtime: no change.

Function main: Temporary variables of type "timeval" change to "_timeb" (and can be largely eliminated). "gettimeofday" is replaced by "_ftime_s". "select" or "nanosleep" (depending upon which is used) is replaced by "Sleep" (note the capitalisation).

I compiled the program with MS-Visual Studio 2005 and the Win32 platform development kit. (Note if you are installing this software there are known
bugs in the MS installer which you have to fix manually before you can compile this program).

For the keyboard and mouse tests, I compiled a special version with only took 20,000 samples (instead of 60,000). This was due to the relatively slow minimum timing resolution which would have caused an excessive test time for
these manual tests. This value is changed in a constant in the source code (LOOPITER).

*************************************

The CPU load program was just a batch file which ran in an endless loop. It loaded the CPU to 100%

REM Load the CPU with an endless loop.

:label
goto label


*************************************

The disk load program was written in "VB Script", as "batch files" are not sophisticated enough. This reproduces a similar result as the "bash" scripts used on the previous tests.


REM Computer Load Test.
REM M. Griffin
REM 16-Oct-2006
REM MS Windows vbscript.

REM Test 4 - Load down disk.


REM Open a file to write to.
Dim fso, DiskTestFile
Set fso = CreateObject("Scripting.FileSystemObject")
REM Set DiskTestFile = fso.CreateTextFile("dummydata.txt", True)


REM Loop around, overwriting the same file each time.
do

Set DiskTestFile = fso.CreateTextFile("dummydata.txt", True)

for i= 1 to 2000
DiskTestFile.WriteLine("2006-10-05, " & i & ", serial-number, data1,
data2, data3, data4, data5;")
next

DiskTestFile.Close

loop


*************************************


The "load" programs were started from a separate command line window, and terminated with control-C after the tests were complete.

Tests at different sampling rates were sequentially run using batch files. These were just variations on:

loadmeas3 15
ren tlogdat.csv tlogdat-NL015.csv

loadmeas3 30
ren tlogdat.csv tlogdat-NL030.csv

loadmeas3 45
ren tlogdat.csv tlogdat-NL045.csv

echo 'Test sequence complete.'

 

In reply to dave - I have posted a previous reply on some testing I have done with MS-Windows (that reply hasn't made it through the Automation List system at the time of writing). I hope you can repeat these tests yourself however, as there are still some holes in my data (see below), and it would also be nice to have confirmation from a second source.

In the previous reply I described the changes I had to make to the "loadmeas.c" program to get it to compile with MS-VC++. It was a bigger headache than I anticipated for such a small program, mainly because the Microsoft documentation of their time and sleep functions (and associated types) is very poor. I can paste the entire modified version into a reply here if you wish, or mail it directly to you (you said you would post an address). You may be able to make your own changes from my descriptions however.

I also included in that previous reply copies of the "load" programs that I used. The CPU program was just a very small batch file (doing a loop), and the disk program was a small MS-VB Script program (the batch file language dosn't seem to be sophisticated enough to do this properly).

I hope you can conduct the testing and report the results here. The testing I did was with a beta version of MS-Windows Vista as I didn't have a version of MS-Windows XP to conduct the tests on. The complete set of tests require several hours without disturbing the computer at all during that time, so it's something that is probably not practical to do at work for most people.

If you have a version of MS-Windows XP you can do testing on, that would be great as that is what will interest the most people at this time. It would be good to have something to confirm my results with MS-Vista also apply to MS-XP. I would also of couse, be very interested in any results which show that I am wrong.

I wasn't able to do the network test with MS-Windows, so if you can do that as well, it would be particularly a big help. You may be able to set up a Samba share that MS-XP will talk to (MS-Vista seemed to have problems with this - probably just pre-release bugs).

The biggest problem I had with porting the test programs to MS-Windows was the poor quality of the documentation on Microsoft's web site, as compared to what is available for just about anything else. They seem to be fairly good at instructions that are at the level of "click this, then that", but real technical details seem to either all be wrong, out of date, missing, or just impossible to find. I've been through the pain already, so let me know if you have any problems.

I look forward to hearing about your results.

 

Hello,

This reminds me of something... Some years ago, I have seen a problem with MS Windows where glitches (in video playback) happened due to the *presence* of the network, regardless of traffic (the network cable was unplugged). Disabling the network card fixed them.

I wonder if it could be that?

I guess that would mean that, as well as "low load", the network tests should also include "network driver disabled".

Jiri
--
Jiri Baum <[email protected]> http://www.baum.com.au/~jiri

 

In reply to Jiri Baum - No, I don't think we should disable any standard OS features. The original point was to show what can be done with a general purpose operating system in its "out of the box" configuration. If we were to tweak and adjust it into a non-standard configuration, then we might as well just use an RTOS. Besides, how useful is a computer without networking these days?

I should point out that I wasn't able to do a network load test on the MS-Windows configuration. Hopefully, someone else will be able to do that test for us (with an "no load" test for comparison). It would be interesting to see if heavy network load made a noticable difference. Your observation raises the interesting possibility that it would make the timing irregularities significantly worse. "Worse" in this case may be either greater or more frequent deviations from the nominal timing.

 

In reply to dave - I've sent a reply to your alternate e-mail address. When you get your results you can either post your own summary here, or you can send the files to me and I will process them in a manner similar to the previous results and post the results here. Either way, a summary of your results will be posted here.

If anyone else has any comments or suggestions on this subject, I would be interested in hearing about them.