Partitioning RT and User space.

K

Thread Starter

Ken E.

Hello everyone.

I think I have an idea here based on Jiris previous email, so I'll tell you what I have been working on lately and where I see could be a
potential way to extend the LinuxPLC and add RT functionality to it.

I have been working a lot with RTLinux at a project funded by my employer. Certainly what you say Jiri about user space processes not being able to inherit hard RT is definitely true. I have written some RT code that does all of the aspects of our mcahine control and logic. machine programmers' code interface with my common machine code through predefined callback kind of functions that are written byt he user with C and a machine control oriented API. I have also written some code to scan a simple Compact PCI Industry Pack IO carrier card, etc.

On the HMI front, I have started building a simple HMI using GTK+. Right now I am passing my system parameters to the HMI user space task through a RT fifo. Its pretty simple, I have a structure and I stream it out to the FIFO as a whole entity and copy it to a structure on the other end, and update the data in that structure to the screen. Simple and effective. The problem is that you have to modify these structures every time you want to be able to add another data variable to pass to the user space HMI. Plus, it is only uni-directional and you must take care to clean up things, etc. Bottom line, it is a bit messy and hard to use as a widespread methodology
of data passing.

I am still using the FIFO methodology indefinitely for what static functionality I have now, but I want the machine programmers to have
access to all the global machine variables fom within the HMI code as well as the RT code. In addition, we are starting to do more data aquisition related projects at my company that involve a co-mingled partitioning of real time stuff versus user space related tasks. For instance, in one application we are sampling a ADC card with a sub millisecond kind of way and the internal buffer just aren't cutting it for this application. So what we really need is simply a RT task that would handle the data sampling and copy the data to a RAM buffer. In this case, most of your application would be in user space, dealing with plots and user input,
etc, and you have a very simple RT task that just buffers data.

What I am proposing for my end of things, and possibly as a add on or enhancement of the linuxplc is to bridge the RT side of things with the user space side of things. How do you do this, you ask? I am grasping at this idea, so let me describe it and see if it makes sense:

Lets assume your system needs both hard real time and user space application programming (RT for control, user for HMI, etc). Lets also assume that both memory space threads need read and write access to common data (where data is RAM variables, IO tables, etc etc). At first glance you start thinking about protocols and things of that nature to try and abstract things and make it flexible, but in essence all you really need is a shared RAM space that contain all of this data. This methodology provides you with a means of treating your entire system as one big multithreaded process running several threads (some of them real time and some non real time.) that communicate with some global IO and variables. Lets face it, all a multithreaded process is just a big shared ram section that
schedules different threads based upon a schedule algorithm(s). To the casual machine programmer, the fact that the different threads are executed by different kernels isn't really known, or cared about. Of course, this concept of resembling a single multithreaded process is not entirely true since you cannot make GTK or disk write calls form within the RT processes, etc etc, but I think you get the point.

As far as IO drivers are concerned, they are very simple. The RT threads have direct access to hardware, so you write a thread that scans the IO periodically and transfers the IO data to and from the IO table/images/structures (pick a word) in the shared memory area to be used by either memory space threads. It would be fast, simple, and very effective.

The reason I like this idea is that now you have a basic automation environment (machine control, data aquisition, scada, a combination of all three, whatever ..) . You have access to RT threads if you need them. You have virtual access to IO. You have access to linux threads. You can
program it natively in C if you want to, or you can add another layer of abstraction and use the linuxPlc environment on top of it. I think there
are a lot of people out there that are usign these proprietary packages such as steeplechase and labview primarily because it makes the interface to the IO seamless and because they allow you to do some real time and HMI as well. I know there are people like me who prefer to not relyon vendors and to program in high level languages. I think the architecture I describe here could be the open source glue between IO, HMI, netowrking, RTLinux, and the actual automation code. Of course we could tap into, merge, and/or cross pollinate between environments like COMEDI and others.

to use it you would include the appropriate header files into your code and compile the project. Then you would load the RTlinux modules which will handle any RT threads and IO scanning threads that you might have defined for yor project. Then you run your user space process which opens up the shared memory area and maps its IO variables into it, etc.

Let me know what you guys think about this idea.

~Ken



Jiri wrote:
********

A few observations as far as relevance to MAT is concerned:

- User-space tasks can't inherit hard-RT priority, so they cannot
interface to the same MAT core as the RT tasks.

This would mean that most likely the computer would end up running two
separate MATs - one in RT doing the control, and another in userspace
handling lower priority tasks, with some sort of protocol between them.
(For instance, it'd probably be unrealistic to have the GUI real-time.)

- The situation with porting SysV IPC calls to RTLinux was not cleared =
up.
However, it's not likely to be a major problem.

- Licensing was not cleared up (at least to me), needs further
investigation.


Jiri

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

 
C

Curt Wuollet

Hi Ken,

Been there, done that, ran it up the flagpole and nobody saluted. The very early work for LPLC,
(demo is in the archives somewhere) used a megabyte of memory, enough for lots of registers,
buffers, etc. that was accessable to both RTLinux and mmapped into user space. The idea was that it was as good a way as any for the present and that we would almost certainly need to use RTlinux at some point. It's based on work done by Fred Proctor of NIST and was, I thought, an elegant solution to quite a few problems. It is reserved from the mm by simply passing 1 mbyte less than your total to the kernel through lilo. This means that it survives the coming and going of processes, even the parent. You simply define the same struct in RT and userland and you have none of the limitations of the fifo method. You have random access and since the userland stuff and the RT stuff do not execute at the same time,
synchronization is simplified, perhaps to the point where we don't need sysV IPC's. Since they don't access the map at the same time and the map is immune to overruns, being parallel in concept
rather than serial, the speed difference for realtime display and slower processes is
handled rather neatly. This map can be any reasonable size, configured at boot and if a regular standard structure were used can be expanded by changing the boot line and adding another instance. This could also be mapped into battery backed NVRAM or perhaps could be a mapped
file for non-volitility. It sounds very much like what you are proposing. Why don't you take a
look and see. I am using this method, albeit with both halves in userland at present, in
production and it is working out well. Mail me off list if you don't want to search the archives.

Regards

cww

--
Free Tools!
Machine Automation Tools (LinuxPLC) Free, Truly Open & Publicly Owned
Industrial Automation Software For Linux. mat.sourceforge.net.
Day Job: Heartland Engineering, Automation & ATE for Automotive
Rebuilders.
Consultancy: Wide Open Technologies: Moving Business & Automation to
Linux.

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
M

Mario de Sousa

Hi Ken,

Thanks for your thoughts!

Actually I think the current architecture of the MATplc (a.k.a. LinuxPLC) already does most of what you describe. Let me briefly explain:

MATplc is based on having several modules running simultaneously. Each module runs in an independent linux process (*1).

The global plc state is stored in two memory areas. One is managed by the cmm library (configuration memory manager), and the other by the gmm library (global memory manager). The cmm stores configuration memory such as the location of plc points, the pid of running modules, etc...
The gmm stores the state of plc points.

To implement what you sugest, we should either:
- go through the current MATplc code, and make it compilable for both
user-space and RTLinux processes;
- write a second MATplc library, using the same memory structure
already used by the matplc, but for RTLinux processes.

I would favour the second approach, as I believe you do to?

We currently use sysV shared memory primitives, but this is very easy to change without affecting the rest of the MATplc code, as the shared
memory code is mostly packed inside a single file /lib/util/shmem_util.c


Would you be willing give a go at writing the MATplc library for processes running on the RTLinux side?


Sorry, got to run. I'll continue this discussion later...

Cheers,

Mario.


Notes:
(*1): Linux threads are implemented as linux processes with shared memory, so going multi-threaded or multi-process does not make much of a
difference umder Linux.

----------------------------------------------------------------------------
Mario J. R. de Sousa
[email protected]
----------------------------------------------------------------------------

The box said it requires Windows 95 or better, so I installed Linux

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
Ken:
> I have been working a lot with RTLinux at a project funded by my
> employer. Certainly what you say Jiri about user space processes not
> being able to inherit hard RT is definitely true.

However, Robert pointed out that LXRT does user-space hard-RT. Unless there's some reason to the contrary, that's a good argument for LXRT.

> I have also written some code to scan a simple Compact PCI Industry Pack
> IO carrier card, etc.

Any chance of contributing this code to the GPL pool?

> Its pretty simple, I have a structure and I stream it out to the FIFO as
> a whole entity and copy it to a structure on the other end, and update
> the data in that structure to the screen. Simple and effective.

Yup.

[using a shared memory buffer instead]
> At first glance you start thinking about protocols and things of that
> nature to try and abstract things and make it flexible, but in essence
> all you really need is a shared RAM space that contain all of this data.

Yup, there's a shared RAM space in the middle of the MAT PLC core... But it also makes sure things don't interrupt each other mid-update. You need to
be careful with that - like I wrote in response to Curt, if you aren't, you'll end up in trouble eventually.

As long as it's just HMI reading, I guess it doesn't matter so much, since all you'll get is a glitch on the screen that will disappear before anyone sees it; but once you start doing anything more than that, you need care.

That's why encoders use gray codes.

> As far as IO drivers are concerned, they are very simple. The RT threads
> have direct access to hardware,
...

Yup - but you don't need to be a RTLinux thread to have direct access to the hardware. Root can do that from userspace. The only thing that can't be done from userspace is interrupt handling.

Jiri
--
Jiri Baum <[email protected]>
http://www.csse.monash.edu.au/~jiribvisit the MAT LinuxPLC project at http://mat.sf.net

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
Curt:
> a megabyte of memory, enough for lots of registers, buffers, etc. that
> was accessable to both RTLinux and mmapped into user space.
...
> You have random access and since the userland stuff and the RT stuff do
> not execute at the same time, synchronization is simplified, perhaps to
> the point where we don't need sysV IPC's.

FTR, my objection to that is that if you don't have synch, you'll sooner or later end up with synch problems, most likely of the Heisenbug variety.

See also: Therac 25

Jiri
--
Jiri Baum <[email protected]>
http://www.csse.monash.edu.au/~jiribvisit the MAT LinuxPLC project at http://mat.sf.net

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
C

Curt Wuollet

Hi Jiri

It's not that you don't have sync, it's that it's inherently self synching if you do it right. This could be done the same way as present with working copies and synced R/W. The other great
feature is that the drivers don't have to have a top and bottom end and can use h/w interrupts. These are much easier to write although care has to be taken that they complete in realtime. In order to do motion control and other time critical tasks we need to have some facility, I'm just stirring the pot so it can be designed in not added on. This is similar to dual port memory
in concept and the techniques can be much the same. The high speed process can read and write atomically and has priority. One RT task synchs the maps IFF no userland proc is changing
content. That can be scheduled at intervals long for the HS task and short for the userland task. It's an RFC. Nothing more. Network drivers are an interesting issue too. I'm running all userland and can see the issues.

Regards

cww

--
Free Tools!
Machine Automation Tools (LinuxPLC) Free, Truly Open & Publicly Owned
Industrial Automation Software For Linux. mat.sourceforge.net.
Day Job: Heartland Engineering, Automation & ATE for Automotive Rebuilders.
Consultancy: Wide Open Technologies: Moving Business & Automation to Linux.

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
Curt:
> It's not that you don't have sync, it's that it's inherently self
> synching if you do it right.
...
> One RT task synchs the maps IFF no userland proc is changing content.
...

That'll do the trick - but that's not inherent, that's something you have to put on top of the shared RAM to make it work.

Anyway, if RTAI-LXRT can now do user-space hard-RT, that's that whole problem area solved - let's just use it.

Three or four reasons off the top of my head:
- userspace processes have memory protection.
- if RT and non-RT processes are in the same class, you can get priority
inheritance (to avoid priority inversion), so you can use semaphores
between RT and non-RT processes and it'll all work right (with care).
- userspace programs are easier to write, compile, run, debug etc.
- less difference between running MAT as real-time or not.

Jiri
--
Jiri Baum <[email protected]>
http://www.csse.monash.edu.au/~jiribvisit the MAT LinuxPLC project at http://mat.sf.net

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
I'm with you, Jiri.

I don't understand the comment that synchronization is simplified, unless you have
fully cooperative multitasking which is the opposite of what RT is supposed to guarantee. Therefore, you have to have synch. Moreover, you need a guaranteed upper bound on any block of the RT.

Rufus

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
C

Curt Wuollet

I agree that if we can simply use something that's out there, we should. I do like dealing with hardware from the simpler rt kernel as that will get us more drivers. Really, all I wanted was to check out Ken's ideas against what I have been using and get some mindshare on the realtime
issues.

Jiri Baum wrote:
>
> Curt:
> > It's not that you don't have sync, it's that it's inherently self
> > synching if you do it right.
> ...
> > One RT task synchs the maps IFF no userland proc is changing content.
> ...
>
> That'll do the trick - but that's not inherent, that's something you have
> to put on top of the shared RAM to make it work.

But all these things are simple and more important, small. Oops, that's for the embedded discussion:^)
>
> Anyway, if RTAI-LXRT can now do user-space hard-RT, that's that whole
> problem area solved - let's just use it.

Cool.

>
> Three or four reasons off the top of my head:
> - userspace processes have memory protection.
> - if RT and non-RT processes are in the same class, you can get priority
> inheritance (to avoid priority inversion), so you can use semaphores
> between RT and non-RT processes and it'll all work right (with care).
Good, if you don't mind being tied to SysV IPC.
> - userspace programs are easier to write, compile, run, debug etc.
> - less difference between running MAT as real-time or not.

"Normal" drivers are nasty to write and nastier to maintain on a moving target. Pure userspace access is subject to scheduling and can't use
interrupts. Lately I've been pondering if all bussed hardware wouldn't be better hard scheduled. Having a number of simple realtime drivers talking through a single map would save a lot of resources without chaos. They would have to be "trusted" drivers (no memory protection),but that's feasible with a 50 line driver. In any case, most of the code is going to
run in userland. I'll have to research what RTAI-LXRT does for the hardware problem. That's pretty tough sledding for us old hacks. In the meantime I've got ATE code to write. I was hoping for at least a brief respite once the machining cell went live but, it's not to be. Fixing the machining bottleneck caused a testing bottleneck. Sigh....

Regards

cww

--
Free Tools!
Machine Automation Tools (LinuxPLC) Free, Truly Open & Publicly Owned
Industrial Automation Software For Linux. mat.sourceforge.net.
Day Job: Heartland Engineering, Automation & ATE for Automotive Rebuilders.
Consultancy: Wide Open Technologies: Moving Business & Automation to Linux.

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
C
From the RT side you have no conflict, it has exclusive access in it's slot. From the user side you have a private copy as now. With a flag in the map you can sync the copy. The uncertainty is on the user side and is buried in the already prodigious uncertainty on the user side.
That greatly simplifies the problem in my mind.
Or, perhaps I'm just raving again :^) EMC works.
If you've looked at how it's done, I've accomplished my purpose. I'm trying to sell the mapping technique that's easily accesable from both sides and a lot more elegant than fifos and has great potential for solving the state storage requirements that people expect from PLC's. You CS types can handle the details.

Regards

cww

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
> > Curt:
> > > One RT task synchs the maps IFF no userland proc is changing content.

Jiri:
> > That'll do the trick - but that's not inherent, that's something you
> > have to put on top of the shared RAM to make it work.

Curt:
> But all these things are simple and more important, small.

Well, there's not that much to semaphores, either. A semaphore is basically just an unsigned integer, with 0 meaning `in use' and 1 meaning `free'. The only magic is in the special ++ and -- functions:

++ increment the semaphore

-- wait until semaphore>0; decrement it

Both these functions are arranged so that they work right with multiple processes, but that, as you say, is a detail for the CS types to do.

> > - if RT and non-RT processes are in the same class, you can get priority
> > inheritance (to avoid priority inversion), so you can use semaphores
> > between RT and non-RT processes and it'll all work right (with care).

> Good, if you don't mind being tied to SysV IPC.

You'd use the LXRT semaphores instead.

> > - userspace programs are easier to write, compile, run, debug etc.
> > - less difference between running MAT as real-time or not.

> "Normal" drivers are nasty to write and nastier to maintain on a moving
> target.

Not that much difference between a normal kernel driver and a RT kernel driver... that's basically what all modules end up being under RTLinux.

Jiri
--
Jiri Baum <[email protected]>
http://www.csse.monash.edu.au/~jiribvisit the MAT LinuxPLC project at http://mat.sf.net

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
J

Johan Bengtsson

What happens if there is more than one processor availiable? Don't you need syncronisation in that case?

/Johan Bengtsson

----------------------------------------
P&L, Innovation in training
Box 252, S-281 23 H{ssleholm SWEDEN
Tel: +46 451 49 460, Fax: +46 451 89 833
E-mail: johan.bengts[email protected]
Internet: http://www.pol.se/
----------------------------------------

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
C
Hi Johan

Excellent question, us poor folks have never had to worry about it. I'm not sure it would make sense to have an RT proc run on multiple
processors. I need to get a MP box and figure out just what they are doing. I imagine it would get very specific to how the multi-processing works.

Regards

cww

--
Free Tools!
Machine Automation Tools (LinuxPLC) Free, Truly Open & Publicly Owned
Industrial Automation Software For Linux. mat.sourceforge.net.
Day Job: Heartland Engineering, Automation & ATE for Automotive Rebuilders.
Consultancy: Wide Open Technologies: Moving Business & Automation to Linux.

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
J

Johan Bengtsson

If you give me proper directions on how to test it and what to test I could do it at home (2 x pentuim III @ 350MHz), yes I have a pair of oscilloscopes too and could put in a 8255 based I/O card.

There is only one big drawback, I would unfortunately have to reboot my computer for that.... well I'll survive &lt;grin>


Even if the RT process(es) only runs on one processor any userland process would be able to run at the same time I think.


I am sometimes reachable at ICQ #4110659 when I am at home and online...


/Johan Bengtsson

----------------------------------------
P&L, Innovation in training
Box 252, S-281 23 H{ssleholm SWEDEN
Tel: +46 451 49 460, Fax: +46 451 89 833
E-mail: [email protected]
Internet: http://www.pol.se/
----------------------------------------

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
C
Sequence is not necessarily priority.

Regards

cww

--
Free Tools!
Machine Automation Tools (LinuxPLC) Free, Truly Open & Publicly Owned
Industrial Automation Software For Linux. mat.sourceforge.net.
Day Job: Heartland Engineering, Automation & ATE for Automotive Rebuilders.
Consultancy: Wide Open Technologies: Moving Business & Automation to Linux.

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
Curt:
> Sequence is not necessarily priority.

I'm not sure what you mean here?

The idea of serialization is to make sure everything works just as if accesses were done in sequence; the simplest way is to use a semaphore to make sure that the accesses in fact *are* done in a sequence...

Or did you mean something else?

Jiri

--
Jiri Baum <[email protected]> http://www.csse.monash.edu.au/~jirib
MAT LinuxPLC project --- http://mat.sf.net --- Machine Automation Tools
tlhIngan Hol jatlhlaHchugh ghollI' Hov leng ngoDHommey'e' not yISuD
Never bet on Star Trek trivia if your opponent speaks Klingon. --Kung Foole

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc
 
Top