# Stupid question about shared memory...

C

#### Campbell, David (Ex AS17)

Hopefully there is an easy answer to the following question:

Why does Puffin use shared memory rather than memory mapped files?
The following are the pros and cons I can think of:

a) Shared memory means only one copy of the global PLC array exists in memory (hence lower virtual memory count)
b) Memory mapped files allows for the global PLC array to be shared between machines via NFS (for high speed stuff you would want gigabit network cards and switched hubs), this could have "interesting implications" (thinking of
distributed systems). Also allows for a mixed operating system environment (providing you have the processor endians taken into account).

David Campbell

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

J

#### Jiri Baum

Campbell, David (Ex AS17):
> Hopefully there is an easy answer to the following question:

> Why does Puffin use shared memory rather than memory mapped files?

Mostly there didn't seem to be any reason to use mmapped files.

> The following are the pros and cons I can think of:
...

> b) Memory mapped files allows for the global PLC array to be shared
> between machines via NFS

For one thing we also use semaphores, which don't exist over NFS, and for another NFS really doesn't sound like the right thing for real-time stuff.

For distributed applications, the several lPLCs will communicate using some protocol. I believe Modbus over TCP is currently being written, and other protocols will no doubt surface in time. We might even eventually grow our own, if we see any advantage in it.

(In a simple distributed application, for instance, there might be one lPLC doing the control and another lPLC doing the MMI.)

Jiri
--
Jiri Baum <[email protected]>
What we do Every Night! Take Over the World!

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

M

#### Mario de Sousa

I guess it could become interesting. We could use the existing nfs (or maybe even SMB) protocol instead of using our own.

The current architecture of the linuxplc lib could support mmaped files, we could even do it as an option, but I think we should keep this till later, after we have got more important things working.

Since we don't have semaphores, this should be limited to read-only remote applications.

Another option to support mmaped files is to build a module that simply makes a copy of the local memory to a mmaped file. This way we could make it read-write, but the semantics would not be exactly the same.

I think the main point is that we leave our options open for the time being, and the current architecture does just that.

Mario.

----------------------------------------------------------------------------
Mario J. R. de Sousa [email protected]
----------------------------------------------------------------------------

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

J

#### Jiri Baum

Mario de Sousa:
> Jiri Baum wrote:

> > Campbell, David (Ex AS17):
...
> > > b) Memory mapped files allows for the global PLC array to be shared
> > > between machines via NFS

> > For one thing we also use semaphores, which don't exist over NFS, and
> > for another NFS really doesn't sound like the right thing for real-time
> > stuff.

> I guess it could become interesting. We could use the existing nfs (or
> maybe even SMB) protocol instead of using our own.

I'm definitely against that.

> Since we don't have sempahores, this should be limited to read-only
> remote applications.

Even for that it doesn't work, because it may get an inconsistent image.

> Another option to support mmaped files is to build a module that simply
> makes a copy of the local memory to a mmaped file.

Yes, that would make sense.

> This way we could make it read-write, but the semantics would not be
> exactly the same.

I'm not sure if that could be done, but perhaps yes. Still, I think neither NFS nor SMB are really suitable for realtime.

Jiri
--
Jiri Baum <[email protected]>
What we do Every Night! Take Over the World!

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

D

#### Dan Pierson

"Campbell, David (Ex AS17)" <[email protected]> writes:

> Why does Puffin use shared memory rather than memory mapped files?
> The following are the pros and cons I can think of:

Shared memory was initially recommended by Curt because it is one of the few available interfaces to RTLinux hard real time code. Since then there's been a lot of skepticism how often RTLinux facilities will be needed. Opinions range from "never" to "more often than you
think" but the project as a whole informally decided to defer the question.

IMHO, memory mapped files on a single machine might be a reasonable choice if RTLinux wasn't needed but memory mapped files over NFS sounds like a prescription for slow but highly variable performance to
me.

Dan Pierson

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

D

#### Dan Pierson

> Shared memory was initially recommended by Curt because it is one of
> the few available interfaces to RTLinux hard real time code. Since

OOPS! Looks like I got the shared memory vs. memory mapped exactly backwards for RTLinux :-(

I still think that memory mapped files over NFS is bad news for many to most applications.

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

C

#### Curt Wuollet

Hi Dave

I have been asking the guys to keep mmap()ed storage as an option. The only thing that is really needed from SysV IPC's are semaphores and those could be faked.

Using mmap() would have some advantages over SysV shmem especially in embedded and realtime environments. Two of these are using storage that RTLinux can interact with directly and providing a non-volatile map. I would hope that they are still keeping that in mind as I think embedded compatibility and small footprint are very important.
After all, we are replacing PLC's which are a general class embedded logic engine.
Ideally we should be able to run on PLC class hardware or at least, cost competitive hardware. If you follow the Linux news, embedded Linux engines are becoming very, very important. As these powerful, inexpensive, industrialized platforms become available we would be wise to port to them so that we can provide small footprint direct replacements where generic PC's are not a realistic option. I am trying to
round up funding to develop or purchase a MachZ or Geode or other POC (PC On a Chip) class
machine with the express purpose of creating a lplc system that looks like a PLC and fits where a PLC does. It may be that this is an important deployment method with development on regular PC's. When trying to penetrate a market, one should never ignore the model that makes your competition successful. Acceptance is much easier if it works much the same way, only better. By covering the whole spectrum we can
offer scalability equivalent to machines from "micro PLC" class to mainframe DCS class
in one product. The PLC makers simply cannot compete with that model.

Regards

cww

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

J

#### Jiri Baum

> > Shared memory was initially recommended by Curt because it is one of
> > the few available interfaces to RTLinux hard real time code. Since

> OOPS! Looks like I got the shared memory vs. memory mapped exactly
> backwards for RTLinux :-(

Hunh?

In any case, there will almost certainly be some way to get shared-memory-like functionality in RTLinux, whatever name it goes under.

> I still think that memory mapped files over NFS is bad news for many to
> most applications.

Agreed here.

Jiri
--
Jiri Baum <[email protected]>
What we do Every Night! Take Over the World!

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

J

#### Jiri Baum

Curt Wuollet:
> I have been asking the guys to keep mmap()ed storage as an option.

It's not a problem - there's a couple of routines that allocate / bind to the memory and return a pointer; anything else that returns a pointer will
work just as well.

> The only thing that is really needed from SysV IPC's are semaphores and
> those could be faked.

I'd be interested to know how you intend to fake semaphores...

(Most other parallel-programming primitives can be used to construct semaphores, of course. But I wouldn't have thought faking them possible.)

Jiri
--
Jiri Baum <[email protected]>
What we do Every Night! Take Over the World!

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

C

#### Campbell, David (Ex AS17)

> From: Jiri Baum [SMTP:[email protected]]
>
> Curt Wuollet:
> > I have been asking the guys to keep mmap()ed storage as an option.
>
> It's not a problem - there's a couple of routines that allocate / bind to
> the memory and return a pointer; anything else that returns a pointer will
> work just as well.

A "popular but unnamed operating system" can only support shared memory by using a shared 'memory mapped file'. The original question was posed to see if the OS in question was really as brain dead as I thought it was (I was trying to avoid peoples prejudices from clouding their
responses).

> > The only thing that is really needed from SysV IPC's are semaphores
> > and those could be faked.
>
> I'd be interested to know how you intend to fake semaphores...
> (Most other parallel-programming primitives can be used to construct
> semaphores, of course. But I wouldn't have thought faking them possible.)

Could a segment of the shared memory be used for this purpose?
Are the semaphores used to prevent two processes/threads from writing to the shared memory area or is there another purpose?
Finally is it possible to collect all the semaphore stuff into a single file rather that liberally scattered throught the lib source tree?

David Campbell

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

J

#### Jiri Baum

Jiri Baum:
> > It's not a problem - there's a couple of routines that allocate / bind
> > to the memory and return a pointer; anything else that returns a
> > pointer will work just as well.

Campbell, David (Ex AS17):
> A "popular but unnamed operating system" can only support shared memory
> by using a shared 'memory mapped file'.

No problem.

Actually, I think it fairly recently became even *less* of a problem, because the routines that allocate and manage the shared memory were split
off from everything else by Mario.

And you may wish to ponder the difference between popular' and widespread'. For instance, the common cold is widespread, but that doesn't
make it popular. Diamonds, OTOH, are popular even though they are rare.

> > > The only thing that is really needed from SysV IPC's are semaphores
> > > and those could be faked.

> > I'd be interested to know how you intend to fake semaphores...
> > (Most other parallel-programming primitives can be used to construct
> > semaphores, of course. But I wouldn't have thought faking them possible.)

> Could a segment of the shared memory be used for this purpose?

I can't see how.

> Are the semaphores used to prevent two processes/threads from writing to
> the shared memory area or is there another purpose?

Yes and yes.

Originally, one was used to serialize access to the shared memory. They are now also used for the synch library.

> Finally is it possible to collect all the semaphore stuff into a single
> file rather that liberally scattered throught the lib source tree?

I guess that's lib/misc/sem_util.[ch] - a fair bit of the semaphore stuff is already in there, anyway.

If you need to construct semaphores out of something else on some OS, you should probably put it in a separate file, maybe lib/misc/sem_<OS>.c or lib/port/sem_<OS>.c or lib/<OS>/sem.c as seems appropriate.

Jiri
--
Jiri Baum <[email protected]>
What we do Every Night! Take Over the World!

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

M

#### Mario de Sousa

> Could a segment of the shared memory be used for this purpose?
> Are the semaphores used to prevent two processes/threads from
> writing to the shared memory area or is there another purpose?

We are currently using three semaphore sets.
One sempahore set (with only one semaphore) is used by the cmm (configuration memory manager) to control write access its shared memory.
Another sempahore set (with only one semaphore) is used by the gmm (global memory manager) to control write access its shared memory.
Yet another semaphore set, with a variable number of semaphores, is used by the synchronisation library to synchronise the execution of modules.

> Finally is it possible to collect all the semaphore stuff into a
> single file rather that liberally scattered throught the lib source
> tree?

Currently practically every access to semaphores is done through the /lib/misc/sem_util.h/c files. I say practically, because the intialization code
of the cmm, gmm, and synch libraries setup the semaphores directly because they don't want the SEM_UNDO flag set when signaling the semaphores. We can change these three accesses very easily, and make them go through the sem_util.h/c files.

As you seem interested, I might just as well tell you that, to the best of my knowledge, every access to shared memory is made through the the
/lib/misc/shmem_util.h/c files.

Mario.

----------------------------------------------------------------------------
Mario J. R. de Sousa [email protected]
----------------------------------------------------------------------------

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

M

#### Mario de Sousa

> Currently practically every access to semaphores is done through the
> /lib/misc/sem_util.h/c files.

Let me just add that I recently added a few routines to this file that need some cleaning up. I would probably like to change their interface too. This was done to get the synch library working, but I am now working on cleaning up the synch library code.

Mario.

----------------------------------------------------------------------------
Mario J. R. de Sousa [email protected]
----------------------------------------------------------------------------

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

C

#### Curt Wuollet

Jiri Baum wrote:

> Curt Wuollet:
> > I have been asking the guys to keep mmap()ed storage as an option.
>
> It's not a problem - there's a couple of routines that allocate / bind to
> the memory and return a pointer; anything else that returns a pointer will
> work just as well.

Yes, we've had that discussion.

> > The only thing that is really needed from SysV IPC's are semaphores and
> > those could be faked.
>
> I'd be interested to know how you intend to fake semaphores...
>
> (Most other parallel-programming primitives can be used to construct
> semaphores, of course. But I wouldn't have thought faking them possible.)

My point is that it's all simply software and can be implemented or emulated to run without explicit system support. Faked in the sense that we do not need the "official" POSIX/Linux/SysV IPC facilities if they are not present in a
pared down embedded distribution. At least I hope that's the case, or we might want to consider alternate methods for synchronization. I would like to investigate just what they typically remove from the kernel to shrink it. It might be a moot point as IPC's shouldn't take up that much core. I've got a couple of the latest
offerings from Monta Vista and Caldera, now I wish I had some time. Maybe one of our lurkers has played with these enough to illuminate.

cww

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc

J

#### Jiri Baum

[email protected]
> > Curt Wuollet:
> > > The only thing that is really needed from SysV IPC's are semaphores
> > > and those could be faked.

Jiri Baum wrote:
> > I'd be interested to know how you intend to fake semaphores...

Curt Wuollet:
> My point is that it's all simply software and can be implemented or
> emulated to run without explicit system support. Faked in the sense that
> we do not need the "official" POSIX/Linux/SysV IPC facilities if they are
> not present in a pared down embedded distribution.

OK. Still, you need *some* synchronisation primitive, from the CPU or the kernel.

> I would like to investigate just what they typically remove from the
> kernel to shrink it. It might be a moot point as IPC's shouldn't take up
> that much core.

And most likely it'd be better (smaller/faster) to put the semaphores back into the kernel than inventing our own in userspace.

Jiri
--
Jiri Baum <[email protected]>
What we do Every Night! Take Over the World!

_______________________________________________
LinuxPLC mailing list
[email protected]
http://linuxplc.org/mailman/listinfo/linuxplc