Site Power Outage Due to Underfrequency

Hello again.

Basically, I work at a petrochemical site which has two 25MW Hitachi H25 Gas Turbines (GT 1 & 2), one Alstom Tornado Gas Turbine of 4MW and a Steam Turbine (STG) of 15MW. We are not connected to any grid and we operate in island mode. GT-2 is normally running on Isoch mode maintaining system frequency at 50 Hz, while all other generators are on droop mode. However, it is to be noted that both GT-1 & GT-2 generators are of same capacity.

Before the event, all the generators were in service. GT-1 was running at 19.6MW, GT-2 at 21.6 MW (Iso), Tornado at 4MW and STG at 10MW. We initiated the shutdown of tornado turbine due to maintenance after creating margin in GT-2 and bringing it down to 17MW. However, due to some fault, Tornado tripped and suddenly there was an additional 4MW load on the system. Ideally, GT-2 should have taken the entire load and nothing should have happened, but what happened in reality was that both GT-1 and GT-2 took equal load i.e. 2MW each. GT-1 load went to 21.6MW, and it tripped on exhaust temperature. At that time IGVs were 78% open. GT-2 (Iso) went up to 19.6MW after which its speed started to decrease. Exhaust temperatures also began to rise and finally GT-2 generator breaker also tripped on under frequency. IGVs of GT-2 were only 68% open when it tripped. The load on GT-2 was 20.6MW at that instant. This resulted in site total power outage and an emergency situation. The tripping time of both GTs is virtually the same only separated by milliseconds with GT-1 believed to have tripped first. The time difference between Tornado tripping and both GTs tripping is 15 seconds.

In all this scenario, I am unable to understand following:

1) GT-2 was on Iso and running at 17MW. Why didn't it take the additional 4MW load entirely since it had enough margin?

2) Why didn't GT-2 IGVs respond on additional load and rise in exhaust temperature. During all this scenario, they only went from 62% to 68%?

3) Both GTs were running fine at 20MW and 21MW before respectively, but when sudden load came, both gave away. Why?

4) GT-1 tripped on exhaust temperature but tripping of GT-2 is very queer. There was enough margin in IGVs when frequency began to decrease which should have catered the additional load?

The event and alarm list can be found at the following link for additional info. Any sort of help would be greatly appreciated.

https://drive.google.com/open?id=1DLPulAapA7H06hfzZr-M2R6wxWbjFXSv

https://drive.google.com/open?id=1bqfD9wq8f1lSXE8h1ZciVpYEbbIC6DuW
 
Hi,

I am joining the discussion out of interest and because i believe i have some contribution to make.

1) So why did GT-2 trip?

If GT_1 droop setting was 1% instead of typically 4% it could cause it to take load faster. See my comment for 3)&4)

2) Why did GT_2 IGV not open to max?

Either there is a problem with IGV reference logic or there is some problem with the IGV controls -because it did not open, GT_2 would have gone into temperature control and prevented from taking more load.

If IGV control was not manual or in temperature matching mode it should have opened as soon as load and temperature increased.

Presuming IGV reference did go to maximum then there should be an alarm indicating demand and feedback error. If this alarm came out then IGV control is the problem.

3)&4) Why the system cannot handle sudden load swings?

Gas turbines are supposed to be best able to take load instantaneously but still expect some time lag - some due to inherent machine characteristic but some avoidable ones are: fuel control set up, MW/Hz setting...

I will take a closer look at GE control software to see what other settings can affect load swing handling.

Best Regards,
Moses
 
Dear Makster,

Very nice job in providing details of your plant and the event. It makes it so much easier to comment. I am not going to be able to give you a definite answer, but only some ideas and comments.

Before the event your plant load was around 55 megawatts. It is assumed GT's #1 and #2 are the only machines that could have responded to a loss in load since the Tornado was coming offline and the Steam Turbine can't contribute much if it is operating in some sort of inlet pressure control.

GT#1 operating in droop control most likely did pick up load since frequency dropped.

GT#2 operating in ISO control most likely did pick up load since frequency dropped.

Something to remember about any axial style turbine is that as frequency declines so does turbine speed. Loss in speed of the compressor means less airflow through the compressor which will increase exhaust temperature.

Your island system instantaneously lost approximately 7% of total system generation when the Tornado tripped. Units #1 and #2 responding by attempting to increase load, but due to underfrequency both the turbines lost speed, which increased exhaust temperature and resulted in a trip of the machines.

In this type of scenario it would be interesting to see what type of load shedding scheme you have in place since load shedding is often needed in this scenario to protect the generating assets. I think in your event the machines were attempting to pickup the loss of generation but it does take time for the IGV's and fuel valves to respond. Any trends you have with fast data, at the millisecond level, would be very interesting to analyze.

1) GT-2 was on Iso and running at 17MW. Why didn't it take
the additional 4MW load entirely since it had enough
margin? It had margin based on it being at rated speed, but the loss in frequency it was not operating at rated speed when it tripped on exhaust overtemperature.

2) Why didn't GT-2 IGVs respond on additional load and rise
in exhaust temperature. During all this scenario, they only
went from 62% to 68%? I would suspect it is a timing issue. It takes time for the actuators to move the IGV's and gas valves. The loss of frequency and machine speed may have been too fast.

3) Both GTs were running fine at 20MW and 21MW before
respectively, but when sudden load came, both gave away.
Why? I would suspect the drop of frequency caused exhaust temperature to rise VERY quickly. High speed trends if available would be something to analyze.

4) GT-1 tripped on exhaust temperature but tripping of GT-2
is very queer. There was enough margin in IGVs when
frequency began to decrease which should have catered the
additional load? I suspect the same as noted above.
 
Dear Moses,
Thanks for the reply.

> If GT_1 droop setting was 1% instead of typically 4% it
>could cause it to take load faster. See my comment for
>3)&4)

Where I can find this exact setting in Mark VI? Can you tell me the name of the signal that is supposed to let us know that droop setting is this much?

>Either there is a problem with IGV reference logic or there
>is some problem with the IGV controls -because it did not
>open, GT_2 would have gone into temperature control and
>prevented from taking more load.
>
>If IGV control was not manual or in temperature matching
>mode it should have opened as soon as load and temperature
>increased.
>
>Presuming IGV reference did go to maximum then there should
>be an alarm indicating demand and feedback error. If this
>alarm came out then IGV control is the problem.

There was no command and feedback error. The output and feedback of IGVs, GCV and SRV were perfectly matching. When sudden load came, FSR changed from 39 to 43% instantly but IGVs command did not change from 60.8%.

I can share the dcast files of the trends. Let me know if you can take a look at those.
 
Hi MikeVI,

>Before the event your plant load was around 55 megawatts.
>It is assumed GT's #1 and #2 are the only machines that
>could have responded to a loss in load since the Tornado was
>coming offline and the Steam Turbine can't contribute much
>if it is operating in some sort of inlet pressure control.

Yes Steam Turbine and GTs are connected in combined cycle with both GTs HRSGs providing steam to steam turbine.

>GT#1 operating in droop control most likely did pick up load
>since frequency dropped.
>
>GT#2 operating in ISO control most likely did pick up load
>since frequency dropped.

My question is that why did the frequency drop in the first place? Isoch GT (GT-2) should have taken all the load without affecting its frequency since it was running before on 21 MW

>Your island system instantaneously lost approximately 7% of
>total system generation when the Tornado tripped. Units #1
>and #2 responding by attempting to increase load, but due to
>underfrequency both the turbines lost speed, which increased
>exhaust temperature and resulted in a trip of the machines.

Understandable but again, why did the frequency drop in the first place. Interestingly, before turbine tripping, its generator breaker opened first as you can see from the events. Generator breaker opening should have caused the turbine to go to FSNL. We still cannot solidly determine what caused the turbine to trip. If you can extract something from the events, that'll be great!

>In this type of scenario it would be interesting to see what
>type of load shedding scheme you have in place since load
>shedding is often needed in this scenario to protect the
>generating assets. I think in your event the machines were
>attempting to pickup the loss of generation but it does take
>time for the IGV's and fuel valves to respond. Any trends
>you have with fast data, at the millisecond level, would be
>very interesting to analyze.

We do have an LMS installed with different priorities set but its only associated with a GT tripping and not on Tornado. The time taken by LMS to act is 200ms. But it did not trigger in this event. Apparently the time difference between both GTs tripping was too short.

>3) Both GTs were running fine at 20MW and 21MW before
>respectively, but when sudden load came, both gave away.
>Why? I would suspect the drop of frequency caused exhaust
>temperature to rise VERY quickly. High speed trends if
>available would be something to analyze.

I can share the dcast or csv files of the trends. Let me know if you can work with them.
 
Hi Makster,

I would prefer the .dcaST files if you can find an easy way to share them. I will be glad to look at them and comment. If the capture blocks were properly configured and produce data at the millisecond level we should be able to see better what occurred and at least discuss it further.
 
Makster,

I'm first looking at the GT-B 5.png document (Alarm List). At 15:29:23.632 the Turbine Underspeed Process Alarm went to NORMAL; we can't see when it went to ALARM. At 15:29:23.362 (the same time) the Generator Breaker Tripped Process Alarm went to NORMAL; again, we can't see when it went to ALARM. At 15:29:25.383 And Exhaust Temperature High Process Alarm went to NORMAL, and we can't see when it went to ALARM.... Finally, at 15:29:25.384 the Process Alarm <b>CUSTOMER TRIP</b> went to ALARM--which is <i><b>probably</b></i> what tripped the turbine (from the alarms we can see in the screenshot).

Looking at the 1.png for GT-A, almost the exact same thing happens--except it's 2 seconds different--15:31:43:534 for the <b>CUSTOMER TRIP</b>. But many of the same alarms occurred--and we can't seem much before that. There was an underspeed that cleared, just like GT-B, and an Exhaust Temp High Alarm, and Backup Exhaust Temperature Reference Active Process Alarm(s). The difference here is that the Generator Breaker Tripped Process Alarm occurred AFTER the Customer Trip Process Alarm.

So, I'm going with MIKEVI in that both turbines had a drop in speed because they BOTH accepted load, and the fact that they both took 2 MW says to me that the unit you think was in Isoch wasn't....

If you can post the exported .csv value files from the .dcast files when the trip occurred, we could probably be more helpful. But the files that were provided pretty much say what was above.

Hope this helps!
 
Dear Makster,

Use finder to locate DWKDG - it is by default 5%.

Another possible scenario is your isoch selection was cancelled by either exhaust temperature high alarm L30TXA or by an auto load lower from MW control block.

Will check out the auto load lower signal when i get some time later today.

I don't know what is dcast.

Best Regards,
Moses
 
Hi CSA,
I really value your inputs and was hoping for your reply. I have posted the links for both dcast and csv files in another reply. Event happened on 3rd September and you can see it in following file: g1_triploglivedata_20190903_100000
g2_triploglivedata_20190903_100000
 
Hi,

observation based on GT-B & GT-A events & based on your info

GT-A is on droop control. No information regarding GT-B on droop/iso.

Based on time stamps (recorder time) GT-B is tripped on under speed first, and then GT-A tripped on exhaust over temp.

And also it leads to confusing why other unit GT-A is not tripped. Seems to be issue with both unit time stamps are not synchronized or wrong? As it was very small load disturbance so two GT's usually should take care.

But during this disturbance, if any one GT tripped, then situation become complicate then it leads to emergency situation.

I am expecting GT-A tripped on exhaust over temp during load sharing of 4 MW, and the other GT-B not able to maintain required load and it leads to under speed and tripped.

If we have some trend data of both GT-A & B together we can analyze further.

And also can you tell me the reason for 4 MW turbine trip is due to internal or any external trip?
 
Hi Makster,

I don't know about your site. But from a typical MkV to MkVIe site now in operation, I found that the GE logic has a serious ERROR.

Isoch is de-selected by L90LL as soon as difference between load set point and actual differs by more than 0.5MW - default setting of LK90DB2.

The logic should have disabled this signal when L83PS is False which it is when Isoch is selected.

I suggest you raise this issue with GE!

Best Regards,
Moses
 
Makster,

I would have responded earlier, but I have been traveling and don't have great Internet access nor a lot of time these days. I was on a plane when I had a couple of minutes for my last reply.

I think MIKEVI is pretty good and has already pretty much identified the issues. And, surya, too. It seems to me that something caused the unit which was thought to be in Isochronous Speed Control to switch out of Isoch and be in Droop--and that's why the two units "split" the 4 MW (because both units are the same rating and have the same Droop characteristic). And, then the units both experienced an underspeed (because there was NOT an Isoch unit to hold frequency), and then all sorts of things started happening because of the underspeed (underfrequency). I don't know what signals were captured in the Trip Log files, but usually they don't have much of what is needed to really troubleshoot trips.

As MIKEVI said, when the airflow through a unit decreases while the fuel is held constant or increased then the exhaust temperature goes up--probably close to the alarm/trip limits.

BUT, from the Alarm Display screenshots, <b><i>both</b></i> units experienced Customer Trips--which means something in the plant issued trip signals to both units a couple of seconds apart, which were displayed in the Alarm Displays. Now, we don't know if that's before or after the units actually tripped.

Troubleshooting trips is not that difficult with WorkstationST Alarm Viewer. Just filter out EVENTS and DIAGnostic Alarms and anything except Process Alarms for a short period before and after when you suspect the trip occurred. Then, with the Alarm Display sorted by Recorded Time, with the earliest time-stamped alarms at the BOTTOM of the window, scroll up through the list and find the first Process Alarm that trips the unit. (If you're not sure, use the list of Trips in the Trip Display to determine which signals cause trips.)

Hope this helps! For the next few days, I'm swamped (literally!) and won't have much time to look over the Trip Log files. Everyone needs to learn how to troubleshoot trips--and now is just as good a time as any to start learning. But, you haven't provided enough information in the first post, and I don't know what's in the Trip Log Live files (hopefully MIKEVI has some time to look them over and share his findings with us!).
 
Thanks everyone for the detailed inputs. After many trials and tribulations, I've been able to map out the exact sequence of events which may help you all further. Time stamps shown are our local time but only minutes and seconds should be the matter of concern! So here it goes:

15.28.51 Tornado Tripped
15.28.57 GT-1 Exhaust Temp High Alarm Appeared
15.29.02 GT-2 Exhaust Temp High Alarm Appeared
15.29.08.155 GT-1 Tripped on High Exhaust Temp
15.29.08.155 GT-1 & 2 Generator Breakers Opened
15.29.08.669 Fuel Gas Compressor of GT-2 Powered Off Alarm
15.29.09 GT-2 Tripped on Customer Trip

Couple of things I should add:

1) We have a logic configured on our Power Plant DCS that whenever Fuel Gas Compressor of a GT trips, DCS issues a command to Mark-VI to trip the relevant GT which appears as customer trip. In this case, when both GTs breakers opened, blackout at site was initiated due to which Fuel Gas Compressor Running indication turned FALSE on DCS after which it issued the command to GT-2 to trip. We believe GT-2 was still running after its generator breaker opening and only tripped after the DCS command

2) The instant when both GTs generator breaker opened is exactly the same, down to the millisecond. The frequency and speed at GT-1 at that very instant was 47.446Hz and 94.645% respectively. The same at GT-2 was 47.641Hz and 96.741%. I know frequency of both should be exactly same but this is what trend showed me. Its pretty clear that breakers opened due to underfrequency. Interestingly, GT-1 tripped on exhaust temperature at that very instant but I think GT-1 breaker opened independently and was not linked with turbine tripping

3) Further analysis has shown that when the sudden load came at GT-2, it changed its FSR from 39% to 43% almost instantly while keeping IGVs at 60.8%. This should have enabled the GT-2 to take the load but it didn't happen because GT-1 had also started shedding its load. As a result, exhaust temperature of GT-2 also increased and whole system went to underfrequency.

I just want to know whether Mark-VI can issue the command to open Generator Breakers or not because our G60 relays and exciter definitely did not open their breakers as per their event log.
Also, our GT-1 exhaust profile was already on higher side before the event and we were planning for its HGPI. Is it possible that GT-1 screwed up everything by first taking the load, then shedding it on GT-2 resulting in underfrequency of whole system?
 
Couldn't find any of these signals. Maybe their name is changed in my system. If you give me some description of these signals I'll be able to locate those.

>Isoch is de-selected by L90LL as soon as difference between
>load set point and actual differs by more than 0.5MW -
>default setting of LK90DB2.
>
>The logic should have disabled this signal when L83PS is
>False which it is when Isoch is selected.
 
Dear Makster,

I was able to download the TripLogLive file for GT#2 at the time of the event. Based on what I see from the trend so far GT#2 was not in Isoch mode. I see the machine pickup ~2 mw of load at 13:45:41, frequency at that time was 49.6hz. Frequency continued to decline and the machine did not pick up any more load like it would if it was in Isoch mode. The breaker tripped at ~13:45:53 when turbine speed fell below 95% speed, frequency at 47.6hz. Gas valves and IGV's were following reference as commanded.

I will download and look at the same file for GT#1. I would ask if you could share the actual triplog files for each unit. What you shared was the TripLogLive files which are files collected and saved at each hour. The sample rate for these is 1 second which is very slow when trying to analyze and event like this. The triplog files should be stored in the same general area you found the TripLogLive files. There will be a folder in the Triplog file folder for each time the unit tripped or breaker opened. If you are able to share the Triplog files for each unit they should also have the alarms and events in the file in the proper time range. The capture blocks setup for your unit are quite good in my opinion, most needed signals are available.

But from what I see so far GT#2 was not in Isoch mode like it should have been. If that is the case then it will be another investigation to determine if some logic issue is to blame or if it was an operational error.
 
Makster, based on the info provided this is what I understand so far.

1) We have a logic configured on our Power Plant DCS that whenever Fuel Gas Compressor of a GT trips, DCS issues a
command to Mark-VI to trip the relevant GT which appears as
customer trip. In this case, when both GTs breakers opened,
blackout at site was initiated due to which Fuel Gas
Compressor Running indication turned FALSE on DCS after
which it issued the command to GT-2 to trip. We believe GT-2
was still running after its generator breaker opening and
only tripped after the DCS command.

I agree and the trend from GT#2 shows the same. GT#2 breaker tripped open when unit speed fell below 95%, but the unit then started to accelerate back towards full speed and then was tripped by the DCS most likely.

2) The instant when both GTs generator breaker opened is
exactly the same, down to the millisecond. The frequency and
speed at GT-1 at that very instant was 47.446Hz and 94.645%
respectively. The same at GT-2 was 47.641Hz and 96.741%. I
know frequency of both should be exactly same but this is
what trend showed me. Its pretty clear that breakers opened
due to underfrequency. Interestingly, GT-1 tripped on
exhaust temperature at that very instant but I think GT-1
breaker opened independently and was not linked with turbine tripping.

From the TripLogLive data the breaker opened when the GT speed decreased below 95%, seeing high speed data might show exactly when it tripped due to high exhaust temperature.

3) Further analysis has shown that when the sudden load came
at GT-2, it changed its FSR from 39% to 43% almost instantly
while keeping IGVs at 60.8%. This should have enabled the
GT-2 to take the load but it didn't happen because GT-1 had
also started shedding its load. As a result, exhaust
temperature of GT-2 also increased and whole system went to
underfrequency.

Data provided shows both units accepted load, but only like they were operating in droop mode. GT#2 did not accept load appropriately like it was in Isoch. With no machines in Isoch mode, there was no way to correct the low frequency.

I just want to know whether Mark-VI can issue the command to
open Generator Breakers or not because our G60 relays and
exciter definitely did not open their breakers as per their
event log.

Yes, if your logic follows standard GE logic then once the signal L14HS(Operating speed 95%) drops out then the MARK system will issue a trip to the breaker 52G.

Also, our GT-1 exhaust profile was already on higher side
before the event and we were planning for its HGPI. Is it
possible that GT-1 screwed up everything by first taking the
load, then shedding it on GT-2 resulting in underfrequency
of whole system?

Based on data provided there was no unit in Isoch mode, GT#1 did not cause the issue. GT#2 did not respond like a unit in Isoch, and it appears it was not in Isoch mode as was assumed.
 
Makster,

The typical logic signal for Isochronous Speed Control selected and active is L83SCI, but it's NOT usually included in the Data Capture blocks for Trip History data. If you have a GE Historian, it might be in that historical database. It might also be an EVENT, but not always. If it was an EVENT, you could filter the Alarms for EVENTs and see if if changed during the situation.

But, MIKEVI is correct--neither unit was on Isoch at the time they tripped. The very fact they both took half the 4 MW load says both units were on Droop Speed Control. You should be looking for what canceled Isoch Speed Control before the trip occurred, because neither unit was in Isoch when the trip occurred--whatever it was that cause the "trip." GE has an extremely bad programming practice of not blocking trip alarms AFTER something has tripped the unit. Again, as has been said repeatedly--when axial compressor speed decreases when frequency decreases the air flow through the unit decreases. If the fuel flow remains the same--or increases because the unit is sensing the speed decrease and is increasing the fuel because it was (THEY WERE) on Droop Speed Control--the combination of decreased air flow AND increased fuel flow will cause the exhaust temperature to increase, resulting in the Exhaust Temperature High alarm. (DLN combustor-equipped units usually operate VERY CLOSE to the exhaust temperature control limit when they are at Part Load, and even if the unit doesn't have DLN combustors but is exhausting into an HRSG the IGVs are usually kept closed to maximize exhaust temperature to maximize steam production at Part Load.

You can't MAKE the cause of the trip be something it isn't--and we really don't know exactly what caused the trip. The Alarm Displays you provided were not complete, and the Trip Log Live data doesn't include alarms. You need to provide the Trip History data for someone to be able to analyze the full alarm list. If you provide the full and complete alarm information and the Trip History data then someone might be able to provide more detail than has already been provided.

The Mark VIe has some excellent tools, and sometimes they are configured well enough to provide very good data and information. In this case, it's necessary to know all of the alarms and EVENTS, and it certainly seems like there was a WHOLE LOT going on with the underfrequency and and underspeed and other DCS stuff going on. Single-shaft heavy duty gas turbines don't always behave as one thinks they should when underfrequency and underspeed is occurring. GE has some poor programming practices that don't block trip alarms after the unit has been tripped--so the Customer Trips (L4CT) may not have been the actual reason for the trip, something else may have already tripped the turbine, but GE's poor programming practices make it difficult to analyze. Not impossible, but difficult.

There are also breaker-open events that don't trip the turbine but do open the generator breaker (such as reverse power). It's possible the generator protective relay(s) tripped the breaker and the unit tripped on exhaust overtemperature. Or the units were tripped by the DCS (L4CT) after the generator breaker tripped.

Help us help you!
 
>Hi Makster,

I sent below message on 13Sept - is this a possible cause of dropping out of Isoch on your site?

"
>I don't know about your site. But from a typical MkV to
>MkVIe site now in operation, I found that the GE logic has a
>serious ERROR.
>
>Isoch is de-selected by L90LL as soon as difference between
>load set point and actual differs by more than 0.5MW -
>default setting of LK90DB2.
>
>The logic should have disabled this signal when L83PS is
>False which it is when Isoch is selected.
>
>I suggest you raise this issue with GE!
"
Best Regards,
Moses
 
Top