Zeroed Starts Counters and Total Fired Time Mark V

C

Thread Starter

ControlGT

Hi sirs.,

One morning I went to check list in the system and run into the values of starts counters and Total Fired Time zeroed in the Cimplicity Screem.

Was not performed any maintenance on the Mark V; the turbine is in conservation; there was no atmospheric discharge.
As graphics, no instrument stopped operating at the time that happens this erasure of the counters.

Has anyone ever had an experience like this?
What could have happened?
How do I retrieve the starts counters and Total Fired Time (data) that were before?

TCI v01.08.03C
TCIMB v02.06.01C
CIMPLICITY 4.0
 
ControlGT,

Are these the only two values that were zeroed out?

Has anyone done any EEPROM downloading to the Mark V turbine control panel control processors and re-booted them recently?

Has anyone made any changes to the CIMPLICITY project or displays recently?

Does anyone open the Mark V turbine control panel processor doors and use a powerful LED flashlight to check on the cards? If so, are all the PROM labels properly affixed to the socketed chips on the control processor cards?

The EEPROM is non-volatile, and about the only way to "erase" it's memory is to do so by overwriting or FORMATting the contents. If a control processor is re-booted a LOT that can cause the RAM to degrade, but that shouldn't affect the EEPROM memory where the timer/counter data is stored.

If someone downloaded ALL or FORMAT to all three control processors and then re-booted them all at the same time that could cause a problem. Is it possible the Mark V turbine control panel lost power for a brief time and then started back up again? I believe the values of timers and counters are temporarily stored in RAM and periodically (once per hour, I believe) transferred to EEPROM for permanent storage. If power was lost to all three control processors at the same time--at just the "wrong" instant in time--it's possible that the memory contents could have gotten jumbled.

If it's just one or two of the timer/counter values that have gone "missing" it's probably just a RAM problem with the HMI. Have you tried re-booting the HMI to see if that resolves the problem? Or, if someone's been making changes to the CIMPLICITY project or the Mark V unit-specific files and not properly copying the information and updating the CIMPLICITY project that could be a possible cause.

As for retrieving the contents, not really likely a possibility.

This is why back-ups are so necessary. And, when performing back-ups, as one of the first parts of the process one needs to upload the timers/counters data to the HMI using the EEPROM Downloader. Then perform the normal process of backing up.

Please write back to let us know what you find, or to provide more information. Again, if it's just one or two of the fields on the Timers/Counter display, it's probably an HMI "issue" and re-booting the HMI will likely fix the issue. If someone's been mucking around and making changes without understanding the process--possibly from another HMI.?.?.?--then that would be a self-inflicted problem that should have prevented by using site-specific SOPS (Standard Operating Procedures) for making changes, and making back-ups before making changes, and limiting access to the HMIs to qualified personnel. (I'm not saying this is what happened at your site--only that is has happened VERY frequently on Mark V sites and recovering can be very difficult without proper back-ups.)

And, finally, even if it was a EEPROM issue, the only way to change the values of timers and counters is to open a PAC case with GE and send them the values and they will return a downloadable file with instructions on how to get the new values to take effect. (There may be some people around who can change timer/counter values, but if so, they probably have questionably obtained the utility from GE. Or, they have reverse-engineered the utility, in which case it may not be certain that it works properly for all versions of Mark V software/PROMs.)
 
just want to know that the counters went 0 or #### in the HMI?

those counters are features of the eeprom, so they are counted there and send to HMI directly, go to the workbench of your project > point > and search for ACCUM_FIRED and ACCUM_HOURS points (those two point that you are visualizing in the HMI) > right click > control panel, and verify the value of each one of them, else try double click and verify that "data type is UDINT" and equation in the virtual section is correct.

start counters and total fired time are used for maintenance puposes and also for scheduling turbine inspections.

Hope that helps, and i wait for your feedback about your problem
 
CSA,

It is UNTRUE that the only way to change counters is by PAC case to GE. There is a somewhat simple but tedious process where you can write to counters from a primitive and change counters/timers. I have done so several times over the past 15 years. All it takes is a little creative thinking and patience. At least one 3rd party can change them with a reverse engineered tool. Also, you are defaming highly qualified 3rd parties within the industry who have reverse engineered communication protocols and created tools which are superior to the GE tools which you accuse them of "have questionably obtained". Note that knowledge of a system architecture contained in one's brain developed from many years of work experience is not questionably obtained.

>And, finally, even if it was a EEPROM issue, the only way to
>change the values of timers and counters is to open a PAC
>case with GE and send them the values and they will return a
>downloadable file with instructions on how to get the new
>values to take effect. (There may be some people around who
>can change timer/counter values, but if so, they probably
>have questionably obtained the utility from GE. Or, they
>have reverse-engineered the utility, in which case it may
>not be certain that it works properly for all versions of
>Mark V software/PROMs.)
 
mhwest,

"...simple but tedious..." is right. Yes; it is possible, but it is tedious using PRIMITIVEs, which is why I didn't mention it. (Will you provide the document to detail the method?) And I have been asked by two sites in the last couple of years about questionable counters which were modified by third-party companies who insisted it worked for other versions of PROMs, but didn't work for those Customers' versions. (PROM version shouldn't matter when it comes to timers/counters. But it sounded good at the time.)

I will rephrase my statement: In my personal opinion the most reliable method to change timer/counter values is to ask GE. Because, if it's critical to get it right the first time, instead of eventually asking GE after several tries, or even one, why not go to the OEM in the first place? Both of the sites mentioned above went to GE after a lot of back and forth and several tries. (Hence the reason for my original statement.)

As for questionable possession, the utility for changing timers/counters was very closely held and protected. I only know of one field person who had a copy, and I wasn't told by him until several years ago. He wouldn't share it. With today's intense concentration on fired hours (equivalent or otherwise) and starts and trips for LTSA's it would be a powerful tool. I've been personally asked to change timers and counters because someone felt, "That WAS NOT a trip!"

My statements were not meant to belittle anyone--rather to save people some aggravation. Hopefully my rephrased statement will be better understood.
 
Mr. CSA,

Thank's for your help!
Below are the answers of the questions:

1- The values zeroed(0000) are: Total Fired Time; Manually Initiated Starts; Total Starts Counter; Fas Load Counter; Fired Starts Couter; Emergency Trips Counter.

2- We've done downloads, but it was 03 months ago.

3- We did not carry out any changes to the Cimplicity project.

4- It was not used any LED flashlight on the cards.

5- We perform tests on the rectifier PEECC batteries, however, before testing the processors were turned off one by one.
The history of DCS, the counters zeroed after normalization of testing the batteries.

6- A few days after I sent the first post, I restarted the Mark V and the HMI, but the values are still cleared. Our machine is in standby, but last week operated for some time empty, and this time was recorded by the counter.

7- On the same day I found the reset counters, restart the HMI, but not normalized

8 - It's a good chance mr. CSA, someone may have sent the old information PEECC, where the local HMI and contains old information. However just me and another employee have authorization to work in qualque MarkV, moreover, only he and I know how to work with Mark V in the company.

Unfortunately I have no procedure or the software to change the counters. So I think I will have to look for GE.

If someone has a safe technique, I accept use.
 
Thank's mr. Isulamu for your help.

The counters were showing the number zero (000).

I found especificamentes the ACCUM_FIRED and ACCUM_HOURS points, I found a number of ACCUM_01_MSW and ACCUM_01_LSW.

Could me rephrase his explanation, please? Maybe I have not looked properly.

<b>Moderator's Note:</b> "especificamentes" may mean specifications.
 
ControlGT,

In the Mark V turbine control panel the recorded values of timers and counters are incremented in RAM and stored on EEPROM. If I recall correctly, the timer/counter values are periodically written from RAM to EEPROM approximately once per hour. I believe the value you see when you look at a display on an HMI is the value in RAM.

RAM is volatile--that means when power to the RAM is lost, the contents disappear. In the Mark V, during boot-up of a processor the RAM is empty, and as part of the boot-up process it has to go to EEPROM to get information (CSP; Control Constants; Timer/Counter Values; I/O Configuration Values, etc.). And, then once the transfer of information from EEPROM to RAM is verified, the microprocessor in each processor (<C>, <R>, <S> and <T>) uses the information in RAM to control and protect the unit. And, then, periodically, (approximately once per hour) the RAM values of timers and counters are written to EEPROM.

So, if the processors were shut down, and then they were re-started and they all came back to A7 without any issues it seems the EEPROM contents were all good, right? Because if all the processors (and their respective cards) went to I/O Status A7 then it would seem the information transferred from EEPROM to RAM was all good. And, that would include the timer/counter values.

What, precisely, was the process that was performed during the shutdown and re-start of the Mark V? It is NOT necessary to do any downloads during a re-start--again, the necessary information is stored in EEPROM, which is not volatile, meaning that even when power is lost the information stored in EEPROM remains and doesn't evaporate. So, if all the other EEPROM partitions were good, why would only the timer/counter values go to zero?

Now, once the Mark V control processors (<R>, <S> and <T>) are all communicating the non-designated processors both look to the designated processor for the correct value of timers and counters and then they synchronize the values of timers/counters in their RAM with that in the designated voter's RAM. While it doesn't seem very likely, there might be a scenario where, for some reason, the designated processor's RAM was not properly populated with timer/counter values from EEPROM and during boot-up it's values were taken by the other two processors. This doesn't seem very likely, but with software and bits and bytes, sometimes strange things do happen.

There are no batteries on the DCC/SDCC cards which might fail over time and lose important information--there's only EEPROM where important information is stored. And, the processors run off the information in RAM. (That's why when most changes to the Mark V CSP or I/O Configuration or permanent changes to Control Constants are made--it's necessary to download to EEPROM and then re-boot the processors to get the information from EEPROM to RAM; the only time RAM has access to EEPROM is during boot-up of the processor--it's just one of the things that protects the Mark V from viruses and hacking.)

To my mind, it's very difficult to envision a scenario where only one section of the EEPROM lost its contents during the power-down time and all the others were just fine. Again, I'm presuming that there were NO downloads made to the Mark V prior to the power-down, or during the power-up (which I really don't know for sure).

I believe you will find that when you get the new TOTD_Q.xxx file from GE it will come with a procedure that will tell you that you need to download the file to the three control processors then power them all off one at a time, wait for a few minutes, then power them up one at a time. In this way the new information which was downloaded to EEPROM will get transferred to RAM, and then all three processors will synchronized their timer/counter values with the designated processor and all will be good.

Hope this helps! GE has a World Wide Web-based "service" they call 'GE Controls Connect' where users can register (by providing some information and a unit serial number) and can then post questions or requests for information or help. This is likely the best way to get their help for this problem, unless you already have a relationship with some division of GE which can open a PAC (Power Answer Center) Case for you. In many cases, the information the provide is free, and I have heard of people getting their timer/counter values reset via this service.

Note that while sites can get help from GE in this way, sometimes GE ask for a LOT of data and information, and even if one site gets help that same information is not available to any other sites. That's one of the GREAT things about control.com--MANY people can see the issues and questions raised by others and also see the responses and usually, read the feedback from the original poster to see if the information was useful or not. GE Controls Connect doesn't provide that ability, or the feedback.

And, just like when posting to control.com for information or help, the quality of the response(s) you receive depends greatly on the quality of the information you provide in your original post, and when asked to provide more information. MANY people believe that their description and assumption of what may be causing a problem are enough for anyone else to be able to understand and troubleshoot and provide a solution for their problem. But, they forget that we aren't there and can't see all of the things they can see and don't know what has (or hasn't) been done to try to troubleshoot a problem. So, the more information you can provide to any forum or individual about the problem, what you've done to try to resolve the problem--and what the results were--the better the response you will receive in the shortest possible time.

Please write back to let us know how you fare!
 
Thanks mhwest for voicing out for those 3rd parties who are really striving to make sure the market isn't being monopolised and to make sure the industry is competitive.

Years ago, I had access to one Mark V panel during my free time and when the panel was available for experiments.

I observed how the change of the counters corresponds the TOTD_Q.xxx file downloaded from the panel, and decoded how the timers and counters are actually stored in the file. I made a small program which translate any designated timers and counters value into the binary values for the 24 accumulators in Mark V. It worked.

However, as far as I know, there should be at least 2 formats in which the TOTD can be formatted, one for PROM version 7.2 and earlier (GASQ and STEAMQ), and another for later versions (DASQ and etc).

I could help you to check if my program will be helpful if you would let me know the CARD_ID of your panel, and a copy of your TOTD_Q.AP1, provided that you are comfortable with it.
 
Again, thank you mr. CSA!

The process carried out before servicing the battery bank:

Before disconnecting the rectifier PEECC, processors were turned off one by one, C, R, S and T.
When the work on the battery bank has been completed, the processors are connected one by one, T, S, R and C; no donwload. All were in A7.

I wondered if the C processor for some storage failure / communication in DCC / SDCC might not be sending the totalization information for the HMI.

I tried to interpret the totalization file, but have not had much success.

The tip on the site "Ge Control Connect" was great, I did not.

I belong to a group of users of the model of our turbine, however, the largest discourses of this group are on electrical maintenance, mechanical, operational processes and parts. However I will expose the subject of totalizing.

I like to post and read about the issues in control.com, always had good results.

I am very grateful to all people from this site.
 
Mr. Stgoh thank you,

Thanks for your report and experiences.
Currently our turbines are in stand-by, so I think a good idea to use your program and experiences to try to manipulate this data.

What do I need to do?
 
Top