Source code control (was: Sequential Control)

G

Thread Starter

Greg Goodman

> When you say "its a source control problem", you are simply saying
> "it's the customer's problem". It is very rare that the average machine
> builder (or even worse - their subcontractors) can seem to deliver a valid
> copy of the source code to their customers at the end of a project. So what
> should the customer do? Should they assume they may have to scrap every
> program that comes with a new machine and write their own? Why would any
> customer find this acceptable?

Michael,

If you're saying that people who write and deliver custom PLC software for a living typically cannot produce an electronic copy of their work that matches the copy installed in the box, I'd have to say that's both a serious source code control problem and a stinging indictment of the level of professionalism in the industry.

I hope it isn't true. In any other realm of professional software development, it would be considered absolutely unacceptable. In fact,
all of my custom software development contracts (I can't speak for everyone else) specify valid source code as a deliverable; failure to do
so would constitute non-performance.

Greg Goodman
Chiron Consulting


 
J

Joe Jansen/ENGR/HQ/KEMET/US

I agree. Source code has always been a requirement for machines I get, and have never had a problem. When I worked on the other side of the fence, I was also required to deliver multiple copies of the source code (as in 2
hardcopy printouts, and 3 or 4 disk copies. Yes, it was before CD-RW).

I am not sure who or why there would be a problem getting a copy of the source code that was loaded into the machine. Someone had to write it to load it in. As long as it is in the spec, you either get the code or you don't pay. Never has been a problem for me!

--Joe Jansen
 
B

Brian Lee Mast, P.E.

Here's an interesting example I saw a few years ago. A systems integrator delivered PLC software under a general construction contract to build a water treatment plant. A few years later, the owner wanted to make revisions to the program and discovered that the "coil" descriptions were all missing. The only comment in the documentation was that the integrator claimed copyright and intellectual property rights to the program. Call them (and presumably pay them)if you want to make a change.

Needless to say, the owner was quite angry. The integrator refused to send the description files at no cost. I believe the owner just reconstructed them by following the drawings and logic.

All my future specifications will include language to avert this.

I hope these kinds of "property" claims to ordinary control programming are uncommon in industry!

Brian

Trippel/Mast Consulting LLC
Seattle, Washington
(206) 652-8485
[email protected]

 
M

Michael Griffin

Greg Goodman wrote:
<clip>
>If you're saying that people who write and deliver custom PLC software for
>a living typically cannot produce an electronic copy of their work that
>matches the copy installed in the box, I'd have to say that's both a
>serious source code control problem and a stinging indictment of the level
>of professionalism in the industry.
>
>I hope it isn't true. In any other realm of professional software
>development, it would be considered absolutely unacceptable. In fact,
>all of my custom software development contracts (I can't speak for
>everyone else) specify valid source code as a deliverable; failure to do
>so would constitute non-performance.
<clip>

I not saying that the the problem always happens. I am saying that it does happen, and it is not a rare event. It would be very unusual if this were the only problem to occur in the course of a major project. It is normally not a very serious one either, as the worst result for a ladder program would normally be to either have a few symbols go missing (which may never be noticed), or in the event of the loss of the program in the PLC, you may get a few bugs back which you thought you had got rid of.

If you had worked as a contractor on a project for me and had made such a mistake (unintentionally delivered a slightly out of date copy), I don't really think I would track you down to tell you about it and I wouldn't be holding back any of your money over it. Perhaps you have made such mistakes when working for other people - how would you know?

The problem I am referring to is when entirely predictable human error causes a working graph program to be rendered unreadable. In this
case, the consequences of an error are all out of proportion to the error itself. The "system" isn't fault tolerant.

I have enough relatives working as computer software developers in various parts of the world to know that the sophisticated and expensive source code control systems that software companies use rarely work as well as advertised. There are simply too many holes in any system. You can't view a system as just a software package, you have to include the people and the environment they work in.

I don't want to concentrate on machine builders and their subcontractors as the source of the problem either. The problem becomes if
anything even more difficult when the machine enters the plant. You have to make sure that if the maintenance department has several computers for troubleshooting purposes (and most do), that the copies they have are always up to date. If one of them fixes a bug by altering a sequence, you have to make sure that this copy gets both archived and distributed. There are just too many places here where something can go wrong.



Joe Jansen/ENGR/HQ/KEMET/US" wrote:
<clip>
>Source code has always been a requirement for machines I get, and
>have never had a problem. When I worked on the other side of the fence, I
>was also required to deliver multiple copies of the source code (as in 2
>hardcopy printouts, and 3 or 4 disk copies. Yes, it was before CD-RW).
<clip>

Since you are now on the production side of the fence, think of it this way. Suppose you were doing a process FMEA and you had this situation:

- Probability of occurance - high.
- Severity of consequence - serious.
- Solution - "Tell the operator not to make mistakes".

Do you think any of your customers would find this acceptable? Yet this is the equivalent of saying "it's a source code control problem". Why is this solution acceptable for the people supplying the PLCs, but not acceptable for the people who are using them in machines?




**********************
Michael Griffin
London, Ont. Canada
**********************
 
J

Joe Jansen/ENGR/HQ/KEMET/US

Joe Jansen/ENGR/HQ/KEMET/US" wrote:
<clip>
>Source code has always been a requirement for machines I get, and
>have never had a problem. When I worked on the other side of the fence, I
>was also required to deliver multiple copies of the source code (as in 2
>hardcopy printouts, and 3 or 4 disk copies. Yes, it was before CD-RW).
<clip>

Since you are now on the production side of the fence, think of it this way. Suppose you were doing a process FMEA and you had this situation:

- Probability of occurance - high.
- Severity of consequence - serious.
- Solution - "Tell the operator not to make mistakes".

Do you think any of your customers would find this acceptable? Yet this is the equivalent of saying "it's a source code control problem". Why is this solution acceptable for the people supplying the PLCs, but not acceptable for the people who are using them in machines?


It's completely unacceptable. Which is why I *do* verify every program that comes in. When the integrator is installing the machine, I bring my laptop out and make sure that I can successfully get online and verify the program against what they have given me. For things like WonderWare apps, etc. I re-load the app onto the machine from the CD while they are still there. I make sure they know ahead of time that I plan to do this. By telling them " I am reloading your entire application off of the CD that you gave me before I sign off on the installation", I have had no problems getting the correct version.

"Tell the operator not to make a mistake" is only acceptable if you are not willing to take the time required to properly verify the application.

Ultimately, it will always be a source code control problem. My PLC's memory already has enough in it storing the application and all the data acquisition information in it. The solution is putting checks in place to verify your source code control.

--Joe Jansen
 
T
The only time I have never received the source code on a job I contracted out was when purchasing a mass produced machine tool. In those cases the OEM often will protect their software and set the protect bit on the PLC. On the other side of the fence, just a couple of weeks ago I performed a service call at a pumping station I set up for a customer. When I connected to the PLC, I found that the program had been modified by someone else, and that turned out to be why there was a need for me to make a service call. If the software in the PLC doesn't match the delivered you know that someone was messing with it. In this case it it voided warranty and I issued an invoice for the service call.
 
G

Greg Goodman

> Since you are now on the production side of the fence, think of it
> this way. Suppose you were doing a process FMEA and you had this situation:
>
> - Probability of occurance - high.
> - Severity of consequence - serious.
> - Solution - "Tell the operator not to make mistakes".
>
> Do you think any of your customers would find this acceptable? Yet
> this is the equivalent of saying "it's a source code control problem".

If the system is designed and built in such a way that it has an operator-induced failure mode with high probability of occurance and serious consequences, I'd say there is a design problem that source code control won't fix.

> For things like WonderWare apps,
> etc. I re-load the app onto the machine from the CD while they are still
> there. I make sure they know ahead of time that I plan to do this. By
> telling them " I am reloading your entire application off of the CD that
> you gave me before I sign off on the installation", I have had no problems
> getting the correct version.

In what way is this different from requiring the source code as a deliverable, and in what way would getting the wrong copy installed NOT be a source control problem?

How acceptable would it be to you for a tech to decide that the solution to a problem is to reconfigure the I/O source of a WonderWare tag - in an application that you have verified - without your say-so, without logging it in the shift notes or maintenance log, and without making
sure that the master copy of the application was updated to reflect the changes?

> Ultimately, it will always be a source code control problem. My PLC's
> memory already has enough in it storing the application and all the data
> acquisition information in it. The solution is putting checks in place to
> verify your source code control.

I agree with this completely. Source code control, like specifications, code reviews, problem reporting and change tracking, are a management responsibility. Management sets the standards, management defines the procedures that are supposed to minimize the incidence of error,
management monitors the system for compliance, and management takes the rap when mistakes go undetected and uncorrected. That's why they get
paid the big bucks.

from one of your earlier emails:

> I have enough relatives working as computer software developers in
> various parts of the world to know that the sophisticated and expensive
> source code control systems that software companies use rarely work as well
> as advertised. There are simply too many holes in any system. You can't view
> a system as just a software package, you have to include the people and the
> environment they work in.

True. Source code control (and all the rest of those management functions) do not consist entirely of technology. It is also political, in that it involves people acting and interacting according to agreed-on rules. Procedures count for more than tools. You can do effective
source code control with diskettes, a labelling convention, and conscientious people; you can do an abysmal job of source code control with SourceSafe on a fully tricked out NT server if there is no oversight and nobody knows or cares what the rules are.

I'm inclined to believe that our areas of agreement are greater than our areas of disagreement. In particular, we both believe in
professionalism and the need to exercise due diligence. I think our area of disagreement here is that you consider control software in the
field to be a malleable thing, and fair game for techs and maintenance folk to make ad hoc changes to. I consider installed software the product of a clearly defined (if cyclical) process that includes design, review and verification. I feel that changes to that software should also be subject to the same rigor. Especially if changes can produce "probability high, consequence serious" modes of failure.

from another of your earilier emails:

> Stop thinking like a software developer, and start thinking like
> someone who wants to operate a plant.

This is probably the root of our difference in perspective. I *am* a software developer, and I don't run the plants I automate. However, I
design the software in the plant, with those who do run them. And I maintain continuing relationships with them, and we continue to maintain and develop the controls software, with due attention to design, verification, source control, testing and documentation.

Regards,
Greg Goodman
Chiron Consulting
 
J

Joe Jansen/ENGR/HQ/KEMET/US

My apologies. My email did not precede quoted text with the > symbol, and you are confusing my comments with Michael's.

I need to ditch Lotus Notes.... <grin>


> Since you are now on the production side of the fence, think of it
> this way. Suppose you were doing a process FMEA and you had this
> situation:
>
> - Probability of occurance - high.
> - Severity of consequence - serious.
> - Solution - "Tell the operator not to make mistakes".
>
> Do you think any of your customers would find this acceptable?
>Yet this is the equivalent of saying "it's a source code control problem".

If the system is designed and built in such a way that it has an operator-induced failure mode with high probability of occurance and serious consequences, I'd say there is a design problem that source code control won't fix.


****Joe****

> For things like WonderWare apps,
> etc. I re-load the app onto the machine from the CD while they are still
> there. I make sure they know ahead of time that I plan to do this. By
> telling them " I am reloading your entire application off of the CD that
> you gave me before I sign off on the installation", I have had no
> problems getting the correct version.

***End Joe****


In what way is this different from requiring the source code as a deliverable, and in what way would getting the wrong copy installed NOT be a source control problem?


----New Joe Comment----

It isn't. That was my point. This simply insures that I get the correct copy. The vendor makes sure because if it isn't, the machine will stop working as soon as I do it. I like to think of it as incentive for the guy waiting for the sign-off before he gets to go home

----end----




How acceptable would it be to you for a tech to decide that the solution to a problem is to reconfigure the I/O source of a WonderWare tag - in an application that you have verified - without your say-so, without logging it in the shift notes or maintenance log, and without making
sure that the master copy of the application was updated to reflect the changes?


****Joe again*****


> Ultimately, it will always be a source code control problem. My PLC's
> memory already has enough in it storing the application and all the data
> acquisition information in it. The solution is putting checks in place
> to verify your source code control.


****end****

I agree with this completely. Source code control, like specifications, code reviews, problem reporting and change tracking, are a management responsibility. Management sets the standards, management defines the procedures that are supposed to minimize the incidence of error,
management monitors the system for compliance, and management takes the rap when mistakes go undetected and uncorrected. That's why they get
paid the big bucks.

from one of your earlier emails:

> I have enough relatives working as computer software developers in
> various parts of the world to know that the sophisticated and expensive
> source code control systems that software companies use rarely work as
> well as advertised. There are simply too many holes in any system. You
> can't view a system as just a software package, you have to include the
> people and the environment they work in.

True. Source code control (and all the rest of those management functions) do not consist entirely of technology. It is also political,
in that it involves people acting and interacting according to agreed-on rules. Procedures count for more than tools. You can do effective
source code control with diskettes, a labelling convention, and conscientious people; you can do an abysmal job of source code control with SourceSafe on a fully tricked out NT server if there is no oversight and nobody knows or cares what the rules are.

I'm inclined to believe that our areas of agreement are greater than our areas of disagreement. In particular, we both believe in
professionalism and the need to exercise due diligence. I think our area of disagreement here is that you consider control software in the
field to be a malleable thing, and fair game for techs and maintenance folk to make ad hoc changes to. I consider installed software the product of a clearly defined (if cyclical) process that includes design, review and verification. I feel that changes to that software should also be subject to the same rigor. Especially if changes can produce "probability high, consequence serious" modes of failure.

from another of your earilier emails:

> Stop thinking like a software developer, and start thinking like
> someone who wants to operate a plant.

This is probably the root of our difference in perspective. I *am* a software developer, and I don't run the plants I automate. However, I
design the software in the plant, with those who do run them. And I maintain continuing relationships with them, and we continue to maintain and develop the controls software, with due attention to design, verification, source control, testing and documentation.

Regards,
Greg Goodman
Chiron Consulting
 
G

Greg Goodman

Joe Jansen wrote:

> My apologies. My email did not precede quoted text with the > symbol, and
> you are confusing my comments with Michael's.
>
> I need to ditch Lotus Notes.... <grin>

Color me pink. I should have known that Michael wouldn't leave himself open to so easy a riposte. "Load the WonderWare app from scratch",
indeed. <g> (Which, by the way, is how I deliver projects. Part of the acceptance test is whether the local administrator can (re)install and
verify the delivered app successfully, using the delivered installation and checkout documentation.)

Well, at least I can stand behind my responses to the snippets quoted from Michael's earlier emails.

Greg Goodman
Chiron Consulting
 
B

Brian Lee Mast, P.E.

Here's an interesting example I saw a few years ago. A systems integrator delivered PLC software under a general construction contract to build a
water treatment plant. A few years later, the owner wanted to make revisions to the program and discovered that the "coil" descriptions were
all missing. The only comment in the documentation was that the integrator claimed copyright and intellectual property rights to the program. Call them (and presumably pay them)if you want to make a change.

Needless to say, the owner was quite angry. The integrator refused to send the description files at no cost. I believe the owner just reconstructed
them by following the drawings and logic.

All my future specifications will include language to avert this.

I hope these kinds of "property" claims to ordinary control programming are uncommon in industry!

Brian

Trippel/Mast Consulting LLC
Seattle, Washington
(206) 652-8485
[email protected]
 
A

Anthony Kerstens

It's uncommon, but not unheard of.

I once seen a group of machines with a rung comment saying something to the effect that the logic was copyright and changes were not allowed. When tech support was brought in the tech, in addition to what he was brought in to do, removed all customer modifications from the logic.

Anthony Kerstens P.Eng.
 
P
I have not seen that kind of thing to work very well. First, most contracts do have language to prevent it. Second, in the one case that I worked on (as Integrator #2) some early and/or bootleg copies of the program(s) were found with some of the customer's maintenance folks. That was more than enough to allow the angry customer to make sure that the modification work did not go to the original integrator. I would suspect that often, that kind of attempt to lock down the software will be foiled by the informal contacts that invariably arise during startup of a process system. Now, skid-mount or other types of turnkey systems may be an exception. Also... GE Fanuc has a "Yes the software is really really unrecoverably read protected forever... Activate now??" feature that I have eyed for years but
never had reason to actually try and use.

Paul T
 
G
What you're describing is not just a copyright claim; it constitutes a limited license to use but not modify the product. Copyright and
license are not the same thing. Even open source software, which you are allowed to modify according to the terms of the license, is
copyrighted by the authors. (And open source licenses typically requires that the user/modifier leave the copyright notices intact.)

Most software contracts do specify who will end up with ownership of the code. Wherever possible, I retain copyright to the work I do, and give my clients unlimited license to use the software. Of course, it's not always possible; sometimes I'm modifying code the client already owns, sometimes the client insists on a contract not for consulting services, but for "product of work". And sometimes, it's just not appropriate for me to claim ownership, as when the client provides significant input into the design, or the software implements proprietary information,
formulae, algorithms, etc. received from the client. (In this latter case, the received materials are covered by the contract's "confidential materials" clause, which constrains me from using the material outside the scope of the contract, or communicating the material to others.)

On the other hand, much of the work I do for specific clients is derivative from work to which I already hold copyright. In that case, it is my ethical responsibility - and usually also a contractual obligation - to disclose to the clients what copyrights I hold to the material I plan to provide to them.

Regards,
Greg Goodman
Chiron Consulting
 
pH control is highly non-linear. The process gain can increase or decrease over three orders of magnitude. What RSLoop Optimizer (AB licensed version of ExperTune) will do for you is linearize the controller input so that the PID will see a linear character of the process and
hopefully maintain desired performance throughout the full range of operation. This may work if your process is stationary.

A non-stationary process will change its character within the same operating regime due to chemical or mechanical changes or disturbances to
the process. If your process is non-stationary, the model of your process that linearizes it will become inaccurate and the controller performance
will degrade (become too aggressive or too sluggish). If that is the case, you will want to find an adaptive controller. One that either
updates it's tunings based on real-time process information or feedback of it's own performance, or one that updates a model of the process that is
either used to tune a control algorithm or is used in a Model Predictive Control scheme. The latter is where on-line learning neural networks are often used. There has been lots of academic research in that area (including my graduate research) but I have not been active in that area
and am not familiar with current commercial products. I have heard of Gensym's G2 expert and neural systems, but I am not familiar with their
capabilities (www.gensym.com).

Another company to look into is NeuralWare (www.neuralware.com). They had been bought by Aspen Tech., but I think they are their own company again. They have neural network modeling software that can produce the code that implements the model, but you then have to implement that code somehow in your control system. They may have better tools now. I used it about seven years ago.

Good Luck! I am interested in hearing what approach you select and implement. Please let me know.

Chip Hinde
[email protected]
 
Top