# Warranty on custom automation software?

E

#### Ed Mulligan

I'm looking to this group for some feedback on what is typical across the industry. I realize that there is no standard, but any feedback would be great.

A typical custom control system might be created from PLCs, PC software, databases, and operator interface software. Often these control systems, or parts of the system, are contracted out to various integrators. Like any product the integrator supplies a quote, the software is ordered, and once it is installed the bill is paid. I'm looking for how service on the software is handled. What is billable and what is covered by some kind of warranty, and for what time period? (I don't mean changes to the software due to the customer changing the process or adding requirements.)

I'm interested in feedback from both integrators and end users on a few
points:

1. How is initial simulation/acceptance handled? Is it the responsibility of the customer, the integrator, or both together? 2. What is done about errors in writing the code that are not discovered during checkout and acceptance? Not logic errors in this case but things like the wrong PLC address entered into a tag, or something like that. 3. How are logic errors handled, such as a special case not being handled correctly? This isn't a syntax error as much as the system was programmed to do the wrong action. 4. How do integrators stand behind their work for problems discovered after the system has been in use for some time?

Ed

Speaking for me, not for Starbucks. . .

M

#### Mark Hill

Ed;

As an SI, I go through this problem on every job I undertake.

Obviously, as an SI it is my goal to protect my company against as many "unknowns" as possible. It is also in my best interest to ensure my clients are happy.

I consider it imperative that my clients and I understand the intimate details of every project. The best way I've found of keeping everyone happy is by establishing "Project Milestones" with "goals" and "rewards" for everyone. Rewards are simple, I get paid and the client gets the product. The "Goals" are the difficult part. As we near a milestone event, I usually submit a list of features and products that I'd like to test, with the client actually performing all tests. We actually create a check-list of everything we'd like to check. To ensure nothing goes unnoticed, I make this list as detailed as physically possible, like checking PLC addresses etc. The client often amends my list and sends it back to me for review. After a few paper shuffles, we usually agree on a goal. This process generates TONS of paperwork, but in the end, we both have a mutual understanding of the goal. I usually work through a series of milestones until the end is reached, and the client performs a "Final Acceptance", where all features and products are tested once again.

Now, warranty.
I warrant all my products and services for a period of two years from the date of final acceptance. If my product fails to perform after final acceptance, the client and I review the failure and compare it against the design goals, and the initial milestone check list. If the failure is obviously my fault, then it's repaired at no additional costs to the client. If the failure results from something the client has modified, we discuss how best to resolve the issue and bring the project back into design specs.

The more detailed the milestone scheme, the less chance of something slipping through the cracks.

Mark Hill - President
[email protected]

P

#### Paul T

Wow... really good questions. I've been all three sides of the fence over the last 10 years (SI, OEM and now end-user) so here goes.

SI's should stand behind their work as long as their installed system hasn't been altered. When I worked as an SI, the company I worked for gave a 1-yr warranty on all work (hardware and software). When I worked as an OEM, we gave the same warranty. Mostly, that's a reflection of the fact that most of the electrical components we used were warrantied for one year.

Initial simulation was always done when I was an SI. It was our responsibility, and we wanted to find problems in our shop not the customer's site. We would do a formal walk-thru of the logic and HMI with the customer, validating the FS and the sequence of operations. We had a large switch/light/pot/gauge panel that we would play process on to do that; it can also be done with software and a PC. In some few cases, like retrofits of old systems, we might not be able to do a simulation but then we tried very hard with walk-thrus and what testing we could do to debug the logic and HMI before we went on-site. Those are the bane of an SI's existance, since usually site startup and debug time is limited and is quoted fixed price.

If errors were found after acceptance, we fixed them. Did that when I was an SI and when I was an OEM and now in my plant too.

Logic errors and incorrect programming... we fixed them.

Well, the standing behind the work part... we fixed it. I personally never had a customer come back after more than 3-6 months with a problem wholly related to the initial implementation. For those of you who have been SIs longer... does that happen often? (Not with your stuff of course... just other SIs, I know!)

So Ed, it sounds like you are having an issue with an SI. For what it's worth, here's the other side of the story. I've been in my current position as a controls engineer with a mfrg. company for a little over two years now. Shortly after I got here, I was put on a project to finish a machine. It's an assembly machine with a GE Fanuc 90-30 processor, five servos, about 200 inputs and 120 outputs. The wiring was done by a local electrical company, I wrote a FIX32 HMI, and a local SI was contracted two months before I got there to wite the ladder.

The local SI had written a good sequence of operations, but no FS and no plans for simulation. I asked about that and was told they were too small but that "it's never been a problem." Uh-huh. To make a long story short, they came on-site in November and after 4 weeks the machine was not even moving let alone running. I sat down for a week with the SI's engineer and we did a line by line code walkthru. We found numerous errors (they didn't understand how to use the servo cards, for example) and discrepancies (not all of the capability descibed in the SO was coded in the ladder). They went away over Xmas and came back after New Year's with a program that, finally, worked. More or less.

Two more weeks pass doing debug and trying to implement all machine capabilities. I sat down again with the engineer and SI owner (well, it was a four-person shop including the owner so that was not hard) and we prioritized some 12 or 15 items not implemented or fully implemented. I gave them another 10 days to tackle that list (we needed the machine) and I agreed to buy it off if they got thru the first six on the list. They did get thru the first 8 or 9 and we accepted the machine. We had originally agreed to a two to three week start-up; we paid for that and we negotiated on how much to pay for the remaing six weeks on-site. (We didn't pay at all for the Xmas rework and they were unhappy about that but agreed that it was their issue entirely.) We paid for about half of the excess on-site time and the SI paid for the other half. Finally, something like two months late, we had a useable if not fully capable machine.

Now, because I am (usually) a nice guy and because we had accepted the machine, we called the SI again about two months after acceptance and offered to pay for his services in correcting the remaining deficiencies. That was not a unanimous decision on our part, but we try to "do the right thing" as corporate culture and driving our vendors out of business or making them hate us is not usually the right thing. Fortunately, the SI was booked and I RAN to another SI in the neighborhood, who I paid more money to (T&M plus the learning curve and all that) but who not only implemented all the machine capabilities, but also increased the machine output rate by about 20%! The original code was so badly written that simply by understanding machine control, and perfoming operations in parallel not serial, we got the rate increase. The only thing that we did was re-work the ladder... no mechanical or electrical changes. It cost us $35k or$40k... and that was paid for in under two months by the increase in productivity.

The moral I guess is that if you don't think you're getting what you should be... you probably aren't. We have since developed a good relationship with the second SI, who did a lot more work for us after that, and we have never done anything with the original SI (who I am not sure is still a viable business). Look at where you are in the project and consider cutting your losses if you need to. Good luck... feel free to drop me an e-mail or call at 603 879-3299 if you want to compare notes.

Paul T

R

#### Rick Lamb

Speaking from an Integrator's standpoint.....
A) I have a standard warranty statement and change order clause that I issue with each system proposal, especially with new customers, so we are on common ground from the start.
B) I have a specific "no warranty" statement that I issue when the customer cannot define what they want or how they want a system to work, and especially for T&M projects.
C) I will customize a warranty statement if the customer requests certain specific warranty coverage that is different and/or longer term than my standard. I adjust the price accordingly. As an alternative I may quote a support agreement.

> 1. How is initial simulation/acceptance handled? Is it the
responsibility
> of the customer, the integrator, or both together?

It benefits both parties to have a simulation, runoff, or acceptance test witnessed by both. A checklist/test plan should have been prepared in advance. Signoff means that the system is complete and tested enough that the warranty period starts, not that everything is perfect and the integrator is done. Some level of training for the operators should be involved. Amazing the number of times a system "doesn't work" according to the operator, but yet it works exactly as the customer's project manager instructed us to make it perform.

> 2. What is done about errors in writing the code that are not
> discovered during checkout and acceptance? Not logic errors in this
> case but things like the wrong PLC address entered into a tag, or
> something like that.

Should be the integrators responsibility, covered by warranty.

> 3. How are logic errors handled, such as a special case not being
> handled correctly? This isn't a syntax error as much as the system
> was programmed to do the wrong action.

Catch this by doing a functional specification, sequence of operation, etc defined up front, then testing to make sure it works during startup. If
there was not a formal spec, at least a meeting between the client and integrator with somebody taking notes and publishing them. Somebody present needs to have the knowledge about the system enough to ask all the "what-if" questions. Those special cases need to be defined before the logic is written, not afterward. The integrator needs to know the equipment/industry and/or be working closely with somebody at the client site who does know the equipment and the special cases. A checklist for the factory test can then be prepared.

> 4. How do integrators stand behind their work for problems discovered
after
> the system has been in use for some time?

Varies by the integrator. Go for cheap, and you will most likely have ommissions and pay extra to get them fixed later, if ever. Pay a little more for experience, and you will have less problems later. Ask specifically for a statement of warranty and the integrators change order policy/procedure. I know integrators who go in cheap and make it up on change orders. I know integrators who are higher priced, but have a formal functional specification and design review process which incurrs overhead but protects both them and the client. There may still be changes, but they are known sooner, not when it's too late and the equipment needed to operate properly yesterday.
I even know experienced customers who add a second line item to their original PO for change order hours... up front. They may add a quantity of 100 hours beyond the base quoted fixed price, not to be used or invoiced for except by specific approval by their project manager. But they know no matter what, there could be an oversite on somebody's part, and they might as well plan for the contingencies, as opposed to having to go back for more  later. If everybody does their job, that money gets left over for something else.

Above all - pick an integrator with enough experience that he can plan for what can/will go wrong, and has allowed for appropriate time to meet with you and go over all the "what-ifs". Ask for and discuss the warranty and change order policies, before you place the order. And make sure you allow enough of YOUR own time during the project to have these discussions and thoughtfully review how you (and your operators) want the equipment to work

Rick Lamb
SpecPlus! Automation Consultants
Division of Midwest Technology Ventures, Inc.
www.MidTechV.com

B

#### Bob Peterson

I have been on several sides of this fence.

As an OEM using an SI to provide PLC programming on a system we provided, and working for a SI.

My experience with the support provided by SIs when I worked for an OEM was not pleasant. They were willing to fix their mistakes if I found them and specifically told them what was wrong, then pointed out (specifically) what was wrong from the extensive documentation they had been given to program from. Once I had shown beyond any shadow of a doubt that there was a problem and it was their "fault", they would grudgingly fix it. I was not real happy with this approach, as they had promised they would test and debug it before they turned it over to us. I ended up spending a substantial amount of time fixing their mistakes myself because it was easier then the hassle of getting them to do so.

In one case an end user hired a large automation vender (who shall be nameless) to design the operator interface screens. When I got onsite to
check out the system, I found that almost none of the functionality existed that was described in the 100+ page document that they were given to design the screens from, and most of what was there did not work. The guy assigned to the project was so hapless the plant instrument tech and I decided to just fix it ourself. The plant engineer later told me that the vendor sent them a bill for something on the order of $15,000 for the two or three days their guy spent on site in a futile effort to fix his mess, before we sent him away and fixed it ourselves. I found this to be somewhat humourous as we had offered to do this work for something like$10,000. The plant engineer told me he had turned down our offer because he thought it was so low we could not possible do it for that price, and paid the automation vendor almost \$40,000 for it. (I am a bit fuzzy on the numbers after all these years - but they are in the neighbourhood).

The SI I now work for really does not have a warranty "policy" as such for software. There are just way too many variables in the mix, from faulty documentation, to changes committed by people outside our control, to things that pop up that have to be fixed, even if they were not part of the original scope. We do not like to nickle and dime clients as most SIs seem to do, so we build in a little bit of time (the amount totally undefined) into each project for field startup assistance from our office. Most problems can be fixed over the phone with an end users technician or an OEM's field engineer, or via modem.

However, if we have to go onsite, its billable. The exception to this rule was a case where one of our employees did not do a good job. We did not bill our client at all for this field work, because we accepted responsibility for the errors. In the 11 years I have worked for this SI, I can only recall a single instance of this happening (not a bad track record in my mind). Generally, we do not charge the client for our mistakes, nor for things we can easily fix, even if its not in our scope. There is a point, however, that gets reached where the volume of changes requires additional funds to cover them. This is a judgement call made by the project manager (usually the engineer assigned to the job in our case) and the engineering VP and sales dept.

In reality, the warranty you get is worth nothing, if the SI does not stand behind their work. Sadly, many SIs seem to make their money off of change orders, so they love to get real legalistic about changes. The only defense
against this, and only real assurance of getting support for correcting any problems that pop up is a good relationship between the SI and the client.

Bob Peterson

M

#### Michael Griffin

<clip>
>Like any
>product the integrator supplies a quote, the software is ordered, and once
>it is installed the bill is paid. I'm looking for how service on the
>software is handled. What is billable and what is covered by some kind of
>warranty, and for what time period?

Usually nothing is billable beyond the initial P.O. except for changes to the original project that the customer requests. What constitutes a "change" and what constitutes a "clarification" is subject to argument. A
normal warranty is for any "bugs" which are discovered within one year.
When that one year period starts should be negotiated in advance. It is not unusual in the automotive industry for new equipment to produce a handful of samples (for product validation) and then sit idle for 6 months or even a year until the final customer (the assembly plant) is ready to go into production. This type of schedule is dictated by the need to build enough cars in advance of actual production to do all the hot and cold weather testing in time to correct any product design defects. In this case, there may not be enough parts produced to tell anything about the equipment until real production starts.

<clip>
>1. How is initial simulation/acceptance handled? Is it the responsibility
>of the customer, the integrator, or both together?

Since you really both have to be there anyway, acceptance (we usually call it "run-off") is usually "both". However, who supplies the labour (if required) is subject to negotiation according to what seems
appropriate and convenient for that project.
Your project specifications (issued for the RFQ) should detail how you want to handle this, including run-off quantities and acceptable performance. The "run-off" should provide a realistic indication of what actual production will be like.
We normally have two run-offs. One at the equipment supplier (enough that we know the equipment is acceptable to ship), and the main one after installation in the plant. We may ship the equipment with certain minor problems noted but unresolved.
The quantities used for the run-off will depend upon the cost and availability of the components consumed, balanced against the risk of a run-off which may be too small to tell us anything. Anything less than an hour's production is not much use. Anything more than a couple of day's worth is not very useful either.
This describes an equipment run-off, rather than just purely a software system. Software only projects are fairly rare with us, and would likely be part of a retrofit. We would normally want an equipment builder or integrator to take an entire project, and let them deal with whatever subcontracting is required.

>2. What is done about errors in writing the code that are not discovered
>during checkout and acceptance? Not logic errors in this case but things
>like the wrong PLC address entered into a tag, or something like that.

This is what the warranty is for. However realistically, if the machine was running fairly well for several months and a minor bug was discovered, most customers would simply fix it themselves. The warranty would usually only be invoked if the fix required a significant amount of work, or if there was concern that any changes would cause a problem with the overall system warranty (e.g., there are other issues outstanding).

>3. How are logic errors handled, such as a special case not being handled
>correctly? This isn't a syntax error as much as the system was programmed
>to do the wrong action.

If the machine is doing something it wasn't supposed to be doing, then this is a bug and that is why you have the warranty. I suppose the real question is whether the machine behaviour was properly specified.

There are two poles to this sort of problem. One is where you know exactly what you want, and specify it in exact detail. For example, you may have somone put in a new MMI system and you specify what each screen is to look like and what you want each button to do. The integrator is providing nothing except their labour, and their knowledge of how the development software works. If you forgot something, then it's your fault. I haven't seen too many of these types of projects. This may be more common where someone is acting as a subcontractor to an integrator, in which case this is the integrator's problem.

The other case is where the integrator is expected to provide some engineering knowledge to the project. This is the more common situation which I have seen. In this sort of project, if they forget to handle a special case this is their "bug", and it is a warranty issue.
Something that is important in this type of project is not to over-specify it. That is, provide enough information to give the integrator a clear idea of how much work is involved, and to specify those points which are important to your production process. Also specify the general design guidelines you would like them to use.
Make it clear to the integrator though that you expect them to come up with acceptable means of handling all the faults, exceptions, recovery, calibration, etc. These are then discussed in your design reviews with them and acceptable solutions agreed upon. There are always cases and situations which only become apparent during the course of the project, and you want it to be clear to them that they will be expected to handle these as they come up rather than just blindly following a spec that couldn't possibly anticipate all of them in advance.
The purpose of a review meeting though, isn't for you to "certify" that their approach is correct so that you assume responsibility for it. The purpose of a review is to allow you to apply your special product and process knowledge to their design to help them avoid "obvious" (to you) mistakes, and to make sure that their solutions are ones you can live with from an operational point of view.

>4. How do integrators stand behind their work for problems discovered after
>the system has been in use for some time?
<clip>

If this is ever in question, then you have a problem regardless of what the warranty says. In any normal project, there is usually enough blame to go around for everyone to have a share. An integrator though, usually depends upon their reputation if they want repeat business. They should be able to give you references from other satisfied customers.

As I said earlier, if all you find is one or two minor bugs, you may be better off just fixing it yourself (provided you have the development software and the knowledge to do this). You could otherwise easily spend more time arranging and overseeing the visit from the integrator than it would be to do it yourself.
If you have reason to want the integrator to fix these (possibly because of the amount of work involved), then you need to be able to give them a clear list of what you perceive the problems to be, and some scheduled time on the machine. The latter can often be difficult once a machine is in production.

The customer should always have their original project manager dealing with any custom software issue that is a question of whether a feature or function was left out or implemented incorrectly. That is, you should not have anyone and everyone deciding what is or isn't a "bug" and calling the supplier to ask that it be fixed unless it is a fairly clear and urgent case (e.g., the program locks, and machine won't run at all). The original project manager knows what was or was not finally agreed to (or compromised on) as the project evolved whereas other people will not.

**********************
Michael Griffin
**********************

F

#### Frank Mitchell

I have been programming PLC's for twenty years - most brands in various situations. Every situation is different. The relationship between
vendor and buyer of custom software should be one of a marriage not a one night stand. Generally the male counterpart the buyer says he will respect you in the morning but they rarely do. Lemme give you an example. A small robot shop in Knoxville was brow beaten and cajolled into taking less money for five pick n place robots. They went broke and later the mfg company which had no engineering dept. of their own ran adds in the paper trying to find someone to fix their robots. Eventually they went broke too. I don't know if it was related to the robots. I have seen the story many times. When you are buying from an established company you can bargain down the vendor and he may be around later but custom software companies are usually small and very fragile. It only takes two unprofitable projects in a row to make them go under. Their programmers move away to other jobs.

If you want aftermarket service I recommend that you treat your technology whore like a wife. When a manufacturing line goes down you stand
to lose far more than you made with bargain software.

The real problem in this industry is that the buyer does not know what custom software is worth. He generally is not informed and resorts to probing psychologically until the vendor squeals in pain. Then the buyer is satisfied that he is getting his monies worth.

One protection for small vendors is to type a software spec up first and have the buyer sign off on the spec before the programming is
started. Then later when the buyer starts complaining that all the bells and whistles aren't pleasing him then the vendor can say if it is not in the spec it will cost you extra. The problem with this is that it is almost as expensive to write a spec as it is to write PLC code.

This country has become pathetically selfish and greedy. I've seen some pretty heartless examples of human behaviour.

The old addage "What goes around comes around" applies here more than ever.

I used to estimate the writing of the code and then charge by the hour for on site startup of the software. It worked fairly well.

Buyers have to realize that buying custom software is not the same as buying something off the shelf. Off the shelf products have gone
through several revisions usually and they still don't work. Microsoft included! Custom software is always rev 1.0 And developing it is more
difficult and many buyers can even comprehend.

Frank Michell

R

#### Rick Lamb

Bob ........
> The only defense against this, and only real assurance of getting
> support for correcting any problems that pop up is a good relationship
> between the SI and the client.

Bob, you are 100% correct on this. But how many more formal relationships (like marriages!) work out unless both parties follow some basic ground rules. There's a happy middle ground with integrators, between highly legalistic and no oversight whatsoever. Enforced communication is the key, and it's a two way street. A documented method of project management, and both parties playing by those rules, cures a lot of these "undefined" problems.

> My experience with the support provided by SIs when I worked for an OEM
> was not pleasant. They were willing to fix their mistakes if I found
> them and specifically told them what was wrong, then pointed out
> (specifically) what was wrong from the extensive documentation they had
> been given to program from. Once I had shown beyond any shadow of a
> doubt that there was a problem and it was their "fault", they would
> grudgingly fix it. I was not real happy with this approach, as they had
> promised they would test and debug it before they turned it over to us.
> I ended up spending a substantial amount of time fixing their mistakes
> myself because it was easier then the hassle of getting them to do so.

"extensive documentation they had been given"....You can't just dump 100 pages of stuff in somebodies lap and expect them to know if it's accurate, correct, complete. You need to have a review meeting, go over it, and ask
the "what-ifs". Unless it's an exact duplicate of an existing system, it will most likely not cover all the possibilities, even if it is extensive. Don't assume your integrator can and has read the documentation, especially if it has been passed down from elswhere within their organization or yours. Have a review meeting and make sure! You'll know right off the bat if the
people have gone through the documentation and are prepared to do the job. If not, toss them out and tell them to come back tomorrow - prepared!

"they had promised they would test and debug it" Promises, Promises! Ask for a factory acceptance test, and witness it! Again, it's worth spending a few hours checking out the functionality BEFORE it gets in your plant.
Two reasons:
a) Its a heck of a lot easier for the integrator to fix in-house without 10 people looking over his shoulder in the plant environment;
b) You can refuse to take shipment and cut your losses if it's really bad work. It's a more complicated situation if you let the integrator install it, start it running, and then decide their work is not good.

> .......I found that almost none of the functionality existed that was
> described in the 100+ page document that they were given to design the
> screens from, and most of what was there did not work. The guy assigned
> to the project was so hapless the plant instrument tech and I decided to
> just fix it ourself.....again, I see the word "given" relating to the
> document. If you had a sit-down meeting with the "hapless" integrator
> about a week after you'd given it to him, and discussed it thoroghly for
> a few hours, you would have figured out by then if the guy understood or
> was an idiot. At that point you could have stopped the project if
> necessary, and done it yourself.
>
> The SI I now work for really does not have a warranty "policy" as such for
> software. There are just way too many variables in the mix, from faulty
> documentation, to changes committed by people outside our control, to
> things that pop up that have to be fixed, even if they were not part of
> the original scope. We do not like to nickle and dime clients as most
> SIs seem to do, so we build in a little bit of time (the amount totally
> undefined) into each project for field startup assistance from our
> office. Most problems can be fixed over the phone with an end users
> technician or an OEM's field engineer, or via modem.

I disagree on this point. You CAN define most of the the variables if you'd had enough experience. You've just mentioned a few yourself. Some you can control, some you can't, and most you can at least observe, monitor, and document if/when they occur. In my long career of systems integration, I've seen my share of jobs gone bad, including my own projects. But those weren't caused by new problems, it was 99.99% of the same old problems over and over. A good set of rules by which you manage projects eliminates the chances of these problems happening. And every time I get in trouble or have seen others' projects go awry, it was because people cut some corners or bent the rules.

Adding the project management and supervision/oversight to a project will
cost 5-10% additional in overhead, for the up front meetings and design review. But it can potentially save a lot more in the long run.

> In reality, the warranty you get is worth nothing, if the SI does not
> stand behind their work. Sadly, many SIs seem to make their money off
> of change orders, so they love to get real legalistic about changes.
> The only defense against this, and only real assurance of getting
> support for correcting any problems that pop up is a good relationship
> between the SI and the client.

.....with that good relationship enforced by playing by the ground rules established UP FRONT.
Such as: a defined warranty policy; A defined change order policy/procedure; and a defined project management methodology incorporating a kick-off meeting, design review, and witnessed factory acceptance testing.

Rick Lamb
SpecPlus! Automation Consultants
[email protected]

A

#### Aaron

If it is CUSTOM software for a specific function in your/the industry, insist on copies of all source code and documentation! JMHO! I have had "integrators" go out of business and disappear from the planet...

E

#### Eduardo Manuel C. Cipriano

The success of a project will depend on both sides, initially the all initial documents such as process narrative, interlocking function,
safety functions, control schemes, and others should be requested by the SI from the customer and both parties should agree and check all of
these documents before generating the programs so that in the process of FAT and SAT minor changes can be adjusted because you have started
the project as getting the inital input documents correct and thus project is done well....

[email protected]
Eduardo Manuel C. Cipriano
Sr. Systems Engineer
Systems Engineering Department
Yokogawa Philippines Inc.