Step to Idle and shutdown after crank/ignition/acceleration without alarm on HMI

Hi,

First time posting, have been an avid follower for quite some time but now in need of some ideas.

Running +LM2500 with DLE (PGT25+/2 stage with pip kit upgrade) MkV has had processor upgrade to DS215...ect

Recently had trouble with processor idle time when viewed from DiagC (processor idle time is down to 15% - 10% when running a view2)
Changed R processor and checked Arcnet cabling but no real joy in improving this overall.
Start ups have exhibited slow HMI graphics and even stalls on purge and ignition delay timers but overall no evidence of severe issues when trawling through DiagC for explanations.
Havent stopped/started the TCI yet but this will be the next port of call however I feel this might be fruitless.

Help

Regards
Animal311
 
Animal311,

This is difficult (for me) because there are Mark V turbine controls, and then there are Mark V LM turbine controls, and the latter is really a Mark V in name only because they are pretty radically different from your run-of-the-mill Mark V (TMR or SIMPLEX). And, it's not really clear which you have, but I seem to recall that DLE LMs were required by GE-Evendale to use the Mark V LM (which had faster processors for protection functions mostly). But, there were some early DLE LMs which didn't have the Mark V LM panel, so, really I'm not sure which you have from the information provided in your post.

Anyway, I'm not really a fan of using DIAGC (I KNOW--people hate it when I say that because that's about the only "tool" that's somewhat available for things like this). And, it's because--at least for the run-of-the-mill Mark V (non-LM versions) the file which defined the information which would be available using DIAGC and the scaling for a lot of that information had to EXACTLY MATCH the versions of all the PROMs which were in use in the Mark V the operator was communicating with. And, there was really only one person in GE Salem who could analyze the files with the versions of PROMsets which were in use in the panel and say for certain if DIAGC would work properly or not (and that person would usually return a proper file for use with the PROMsets which were installed in the panel). BUT, there were--and are--LOTS of operator interfaces out there with incorrect DIAGC files. Hence, my reluctance to rely much on the information from DIAGC.... Sorry.

Anyway, my personal suspicion is that the GE Mark V HMI is not it's normal, happy self. A lot of the HMIs used back in the day used versions of MS-Windows that were known to leave LOTS of fragmented files and file bits and pieces on the HMI hard drive which, over time, would considerably slow things down on the HMI. AND, it could even result in excessive requests for data from the Mark V which could also cause low processor idle time.

My first suggestion would be to make a back-up of the HMI (that should ALWAYS be the first thing before doing anything with the HMI, or CIMPLICITY, etc.). Use an image application such as Acronis or something approved by your corporate IT department and make that back-up as soon as possible. (To make a proper back-up, you need to shut down the CIMPLICITY project, and shut down TCI. Shouldn't be a problem; almost never is (notice I said 'almost' because there are zero guarantees in this endeavour). I recommend backing-up to a very large external hard drive (if the HMI CPU can accomodate an external USB hard drive). I have even seen some good IT departments come and install a hard drive in a spare internal hard drive location in the HMI CPU and back-up to that hard drive.

Once the back-up is completed--and while CIMPLICITY and TCI are both still not running, I would then use whatever utility is available in the MS-Windows version on the HMI and defrag the hard drive. I've done this many times with only a couple of problems (and those were both caused by zero "free" space on the hard drive because of very high fragmentation). Once the defragmentation is done (and it could take several hours depending on the HMI CPU and version of MS-Windows), then I would run check the hard drive for any errors using whatever utility is available in MS-Windows. (Many corporate IT departments can help with these tasks, and almost always have some external utility they prefer to use that is almost always faster and better than whatever was supplied with MS-Windows--so if you have access to such a department, and you're on good terms with them, I strongly suggest getting them involved, because if the hard drive is severely fragmented they may be the best ones to help deal with the problem.)

After all that, the HMI will probably have been re-started a couple of times (I would recommend leaving the StageLink (ARCnet) coaxial cable disconnected from the ARCnet card in the HMI CPU whilst this is happening (the back-up, defragging, and error-checking)), but if not, then I would suggest a normal shutdown of MS-Windows, and after it has stopped for a couple of minutes, then re-connect the StageLink cable to the ARCnet card and re-start the HMI normally, letting it start TCI and CIMPLICITY and all that.

If the hard drive was even mildly fragmented, one can usually see a difference in response times and screen update rates, and even an improvement in Mark V control processor idle time(s).

With older versions of MS-Windows (98, XP, even NT) disk defragmentation is pretty critical and should be a part of regularly scheduled maintenance, as should regular back-ups of the HMI hard drive. An ounce of prevention is worth several pounds of cure (which usually come from the arse-chewing one gets when the HMI and/or hard drive "dies" and is unusable and the site is scrambling for fast replacements--which are hard to find and not inexpensive).

I would also suggest, if you haven't powered-down the Mark V (run-of-the-mill or LM) for a while, that's not a bad thing to do whilst all of the rest of the activities described above are happening. Sometimes, it really, really helps to clear out the RAM and dual-ported RAM which was used extensively in the Mark V product line. If there is extreme resistance to powering down the entire Mark V panel, then I suggest doing one processor at a time, waiting for several minutes before re-booting the next processor. AND, I strongly suggest using the switches in the <PD> core for cycling power to the control processors, opening the switch, waiting about 20-30 seconds, and then closing the switch and waiting for several minutes after the control processor has returned to normal I/O status before moving to the next control processor.

Here's another area where I'm not very well versed--rebooting Mark V LM processors. They are something of a different beast; some of them having their own hard drives on the main microprocessor board and using a different real-time OS (operating system) than the run-of-the-mill Mark V turbine control panels. SO, if you know there is a recommended procedure to follow for powering-down and re-powering the control processors in a Mark V LM panel, follow that procedure (NOT the one I outlined immediately prior--as that is for a run-of-the-mill Mark V.)ff

Short of the above, and based on the information provided, I can't think of anything else to try. If you are nervous about defragging the HMI hard drive, then at least run the defrag utility--and see how badly fragmented the hard drive is. Then mull over your decision. But, I can tell you, that early versions of MS-Windows used a LOT of page files and hard disk space as temporary storage in place of RAM (which was expensive back then and which MS-Windows didn't always use very well even if it was available). And, that resulted in lots of problems with hard drives--even when the defrag utility didn't report very much fragmentation. And, this also resulted in a number of bad sectors on hard drives, that were no longer available to the OS and which caused problems for TCI and CIMPLICITY, also. (CIMPLICITY and TCI back then (in the Mark V production days) would open large temporary files and would leave them "open" and unavailable for use if the HMI lost power or wasn't shut down properly, and sometimes even if it was shut down properly (TCI is not the most well-behaved service ever written).

Anyway, that's about all I can think of. Usually, when nothing else has changed recently, and the HMI has been running for a long time (months, or even years) it's hard drive fragmentation that is the problem for sluggish response, and even Mark V control processor low idle time(s).

Please write back to let us know what you find and how you resolve the issue!

P.S. Don't forget--if regular hard drive images (what I call "back-ups") aren't being done, they should be!
 
CSA,

First of all I just wanted to quickly thank you for getting back to me so quickly.
I have to say I didn't expect you to go to such depths with your reply but I'm extremely glad you did.

So before I get into the nitty gritty I thought id give you the details of the system we have.
Our MkV configuration is the Mark V LM, we don't have the TMR or the Simplex, I didn't realise this until a few years ago as I only have experience on my own site equipment, I went to the Florence GE learning centre and quickly began to learn just how complex these systems were and indeed how many different configurations there are, so I apologize in advance because I know how difficult it can be discussing these issues when the other party isn't strictly in the know with the overall picture.

The MkV in question I believe is the LM MkV, it runs 3 x TCEA safety cards in the P core which directly feeds into the R core, I'm pretty sure we have the faster processors (the DS215) but this wasn't the case with the original build back in 1998, they were upgraded in 2003 by GE but this was down to a recommendation they made to my company.

So back to the future... I cant help but agree with your comments about the DiagC, I've always thought it funny how the supply voltages to the cards never correspond to the readings you will take at the test posts on the power supply boards, if I'm completely honest we only really use DiagC for A4-A7 status on the cores or for the idle time data, I've tried teaching myself the other aspects of the programme but it seems to be black magic a lot of the time... so please don't say sorry, you've brought me round to some new thinking.

When this issue with the idle time first manifested I did initially think maybe the issue might be the HMI but I kind of got pushed down the route of MkV because historically any problem we have had normally 9 times out of 10 its down to a card failing or a bad ribbon cable ect ect.

We take regular back ups of the MkV HMI (I think we currently try to do 6 monthly but sometimes these might slip a little), We do happen to use Acronis on the MkV's so I have my arse covered in that respect, either way thanks for the suggestion of doing one before I try tinkering with the TCI or Cimplicity, I think this may have slid under my radar tomorrow after a nights sleep.

The more and more you go into detail with the disk defragmentation the more and more I cant help but feel your bang on the money, I don't think I have carried out a defragmentation on either of our MkV's in my time as tech (3 years) and im pretty sure this may go back 5-10 years so I think the nail may have been hit on the head. We run windows 98 so like you say it is critical and I think this has to be a perfect place to start.

We have powered down the MkV quite a few times in the past couple of weeks, it seems to be behaving quite well with boot ups and such, normally if it has its knickers in a twist this is our go to for a quick fix but we had no joy this time round so this again points to another silver bullet in the corner of disk defrag. Interestingly enough we do have a procedure for powering down the LM MkV which was given to us by a recent visit from a GE MkV specialist/trainer.
You've been nice enough to me so let me share something with you...

We are told to boot the switches in the PD core in this order 1 - R1 core (Q11 also comes on with this, 2 - R2, 3 - R3, 4 - Q51 5 - R then 6, 7 and 8 (these are the TCEA cards in the P core 6 being x, 7 being y and 8 being z).
You would think that the design of the switches goes along with 1 through to 8 but we did once have a GE tech who told us a secret code for the switches if you ever had trouble with boot up, I still have this written down (if you ever require it il be happy to share) but recently we have been told this was a load of rubbish.

I cant thank you enough for the help you gave in your reply earlier, I have always thought of the MkV world as an isolated solitary place, nice to know there is help out there from genuinely good people just looking to offer friendly advice.
I will get back to you as soon as I have any news on improvements or further frustrations.

Thanks a million

Regards
 
Animal311,

Thanks for the information!

Yes; it's an occupational hazard--that most people who have only worked on one GE gas turbine with a Mark* (IV, or V, or VI or VIe) think every other GE gas turbine with a Mark* (IV, or V, or VI or VIe) is the same as the one they are working on.

Some forums have something called avatars, where people create their "avatar" (sort of a user profile) and explain about their experience and the equipment they work on or have worked on and put some details which would be very helpful for those trying to provide help and information. These work quite well, when they are used and when they are "filled out" properly--but they do take time to fill out, and most people are simply interested in getting their problem solved and don't want to take the time to fill out their avatar/profile. It would be nice if this feature were available here on Control.com, and if people were encouraged to create an avatar/profile when they first visit this site. You have said you have been a long time "lurker" and I think a lot of people are similar (though not everyone!). And I imagine (and I have a very vivid imagination at times--more like wishful thinking) if you could have filled out an avatar/profile, even if you never planned on actually using this site for help or information, it would have been very helpful.

Anyway, please write back to let us know how this potential solution works out for you--I for one, am VERY curious.

I am also very please to hear your site has a practice of periodic and frequency "back-ups" (imaging)--that's to be commended!

Lastly, have you considered what you're going to do when (because it's not "if,", but "when!") the ARCnet card in the HMI fails? Do you have spare ARCnet cards? Also, I know that not every new PC CPU can work with MS-Windows 98 (directly) or has PCI slots for network interface cards, so if the CPU fails, do you have a plan? (For example, using a virtualization program--though I'm not sure that would work directly with an ARCnet card, but there might be other options I'm not aware of for this.) Just curious, because, it is an eventuality (component failure).

I look forward to hearing how this plays out (the defragging, at least).
 
CSA,

So firstly we have identified the problem causing our issue with the GG getting up to idle speed setpoint (Step to Idle and shutdown after crank/ignition/acceleration without alarm on HMI).
The issue was down to the VSV diff on the LVDT's, when you look through our logic this is the only step to idle point which doesn't give us an alarm through to the HMI, we had to build a view2T to identify which point was causing the step to idle and then trace it back through the logic but we eventually saw the open circuit value on the position for the north (B) VSV LVDT.
Truth be told we ran a few failed calibrations before we realized the aero connector had worked itself loose however we now have a very happy compressor package..... I say compressor package because we have found you were spot on with the HMI processor theory.

So with regards to your second point we have started to address the issue of component obsolescence with regards to our MkV's.
We have recently (over the past two years) upgraded two of our MkV's to the new MkVe which leaves us with just two MkV's for the time being.
The upgrade has included a raft of improvements however the two big items for me were the HMI upgrade and the change to electronic fuel valves.
The rationale of this decision was mainly down to availability of the older hydraulically driven fuel metering valves, we have over the past ten years nearly exhausted our entire stock of spares for these and it was getting harder and harder to get this overhauled.
With this we now have two full sets of spares for the older MkV's if we ever have an issue with various component's across the package.

Now we are looking deeper and deeper into the processor issue its my belief that we do genuinely have an issue and we have decided to run the defragmentation tool on the HMI however we are going to hold off until we hit our slow period within the year. We have a genuine worry that we could kill the hard drive completely when we do it so we are going to duplicate the hard drive on the unit currently and then at least that way we have a plan for a speedy recovery if something does fail.
In the meantime we are going to run the defrag analysis, when we ran the unit up recently the HMI processor was topping itself out at 100% so we will wait to see what we find from this.

When we run the defrag I will be sure to remember to contact you and let you know how we get on, I currently have a few colleagues who are discussing whether this will cure the issue or not but I'm firmly in the corner of it needs doing and needs doing on a more regular basis.

Thanks for your help on this, I really appreciate you taking the time to reply and share some of your wisdom, if you ever have a scenario were you need some help with an LM feel free to pick my brains, although being the fountain of knowledge that you are on here I'm doubtful this will happen

(y)
 
Top