Mark VIe R controller diagnostic, IONET1 issues

Hi CSA,

Thank you for your detailed response. With the migration philosophy you mentioned, yes it was not a Mark-V to Mark-VIe migration rather a new Mark-VIe where all control system side hardware was replaced with Mark-VIe.

The system is running since the faults first appear and we cannot take it down without taking a power load debit on our facility. The system has both TRPG and TREG boards for control and protection. We checked for ground fault but found 28V supply to be healthy on all IO packs.

We have followed the below steps so far without any success:

  1. Unplug the <R> controller to IONET#1 Switch cable from existing port and plug it in spare port. Diagnostic alarms were not cleared.
  2. Unplug & Plug-in the IONET#1 cable on <R> controller. Diagnostic alarms did not clear.
  3. Replaced the Ethernet cable between <R> controller to IONET#1 switch but Diagnostic alarms did not clear.
  4. Recycled power of IONET#1 switch#1. Diagnostic alarms did not clear.

We plan to do teh following next:

  1. Reboot the <R> Controller by pressing the white button located at the bottom of UCSB.
  2. If alarms are not cleared, we will replace the IONET#1 Switch#1 and check if the diagnostic alarms are cleared though space congestion will be a big issue due to location of the switch.
  3. Lastly if nothing works, we will replace the processor and download the application again.

Another item to note here:

UCSB.T also noticed communication error with R pack IONET#1. However this is intermittent and restores. Fortunately both UCSB.T and UCSB.R errors have not occurred together otherwise I believe Unit could have tripped. But why is IONET#1 healthy even after power cycle? Even the cables are separate for Both R and T controllers on IONET#1 switch.

View attachment 1389
Hi MnM

You stated that:
  • Reboot the <R> Controller by pressing the white button located at the bottom of UCSB.

Are you referring to backup /restore white push button located on UCSB controller..
I strongly advise you to do correct things according to OEM technicals manual..

Please read first carefully the right manual prior to do anything

As you may perform a backup for sure...

James

James
 
MnM,

I am really perplexed. The screenshots and the descriptions seem to me to be somewhat contradictory, but I’m probably missing something.

UCSB.S and UCSB.T have to know what the I/O states are that UCSB.R is receiving. All three processors need to know what the other processors are receiving so they can each vote the I/O states before executing the application code.

I don’t have access to a Mark VIe or the manuals at this writing, so I am unable to comment on the little white button’s purpose and function. RESET doesn’t always mean the same thing, especially when GE is the source of the usage…. To myway of thinking, a power-cycle (also known as a hard re-boot) is the best re-boot. (The Mark V had a little white button on the DCC/SDCC card that people loved to use, but when it didn’t work as expected they were always upset and blaming the Mark V for problems they created because they didn’t understand precisely what the little white button did—and didn’t—do.)

I’m not a swapper when it comes to troubleshooting. Cables, yes. Cards and I/O Packs not so much.

Please seriously consider the testing ControlsGuy25 is recommending. I was not previously aware of the capability, but it might be helpful in pinpointing the problem.

if there is more than one IONET1 Ethernet switch in the IONET1 circuit, you might try seeing if the packs with the issues are connected to a single switch—and replace that switch first.

I wish I had better news and/or recommendations but I would really need to be sitting in front of the ToolboxST at your site to be seeing what you’re seeing and be able to run the diagnostic test(s) James is recommending.

We also don’t know how old the installation is, nor what TILs (Technical Information Letters) and/or PSBs (Product Service Bulletins) have and have not been implemented over the life of the system since installation.

Thank you for keeping us updated on your progress and wish you the best in resolving the problem.
 
MnM, I would suggest that this most likely sounds like an issue with:
One if the IO net switches for the "R" core network.
Power Supply for the "R" core and IO packs.
Issue with the "R" core processor.
With the unit running it is difficult to diagnose the issue without risk of tripping. I am not sure how intermittent this is, and for how long the issue lasts when it is in the failure mode. The longer it stays down the easier it should be to diagnose if you are allowed.
The fact that all IO packs are losing communication with "R" core then it is very unlikely it is an IO pack issue, unless a IO pack is somehow taking down the entire "R" core network, which I guess is possible but I have never seen. I have seen issues with the switches when NTron was the manufacturer, not sure if yours are NTron or GE. I would start by replacing the network switch that is connected to the controller that could cause it to lose connection to other switches and the IO packs.
The white button the bottom of the processor is not a reset, it is for backup and restore purposes.
Please keep us posted on your progress.
 
Hi MnM

You stated that:
  • Reboot the <R> Controller by pressing the white button located at the bottom of UCSB.

Are you referring to backup /restore white push button located on UCSB controller..
I strongly advise you to do correct things according to OEM technicals manual..

Please read first carefully the right manual prior to do anything

As you may perform a backup for sure...

James

James
Hi James,

My understanding was the same that it was a backup and restore button but we were to by BH team (who are providing limited remote support at the moment) to reboot the controller using the white button but unfortunately that did not work in our case and the controller remained in same state.
 
MnM,

I am really perplexed. The screenshots and the descriptions seem to me to be somewhat contradictory, but I’m probably missing something.

UCSB.S and UCSB.T have to know what the I/O states are that UCSB.R is receiving. All three processors need to know what the other processors are receiving so they can each vote the I/O states before executing the application code.

I don’t have access to a Mark VIe or the manuals at this writing, so I am unable to comment on the little white button’s purpose and function. RESET doesn’t always mean the same thing, especially when GE is the source of the usage…. To myway of thinking, a power-cycle (also known as a hard re-boot) is the best re-boot. (The Mark V had a little white button on the DCC/SDCC card that people loved to use, but when it didn’t work as expected they were always upset and blaming the Mark V for problems they created because they didn’t understand precisely what the little white button did—and didn’t—do.)

I’m not a swapper when it comes to troubleshooting. Cables, yes. Cards and I/O Packs not so much.

Please seriously consider the testing ControlsGuy25 is recommending. I was not previously aware of the capability, but it might be helpful in pinpointing the problem.

if there is more than one IONET1 Ethernet switch in the IONET1 circuit, you might try seeing if the packs with the issues are connected to a single switch—and replace that switch first.

I wish I had better news and/or recommendations but I would really need to be sitting in front of the ToolboxST at your site to be seeing what you’re seeing and be able to run the diagnostic test(s) James is recommending.

We also don’t know how old the installation is, nor what TILs (Technical Information Letters) and/or PSBs (Product Service Bulletins) have and have not been implemented over the life of the system since installation.

Thank you for keeping us updated on your progress and wish you the best in resolving the problem.
Hi CSA,

There are two network switches where SW#1 is communication with Analog IO packs, RTD IO packs, Vibration IO packs as well PPRO and PTUR boards while SW#2 contains digital IO board. SW#2 is connected to SW#1 which is connected to R/S/T controllers.

The installation and commissioning of the unit was done in 2018 and generally we have covered most of TILs and PSBs.

Thank you for your support and guidance. Please let me know if anything comes to your mind in this regard.

Regards,
MnM
 
MnM, I would suggest that this most likely sounds like an issue with:
One if the IO net switches for the "R" core network.
Power Supply for the "R" core and IO packs.
Issue with the "R" core processor.
With the unit running it is difficult to diagnose the issue without risk of tripping. I am not sure how intermittent this is, and for how long the issue lasts when it is in the failure mode. The longer it stays down the easier it should be to diagnose if you are allowed.
The fact that all IO packs are losing communication with "R" core then it is very unlikely it is an IO pack issue, unless a IO pack is somehow taking down the entire "R" core network, which I guess is possible but I have never seen. I have seen issues with the switches when NTron was the manufacturer, not sure if yours are NTron or GE. I would start by replacing the network switch that is connected to the controller that could cause it to lose connection to other switches and the IO packs.
The white button the bottom of the processor is not a reset, it is for backup and restore purposes.
Please keep us posted on your progress.
Hi MIKEVI,

We rebooted IONET#1 SW#1 and everything came back online but the error persists. We even swapper the ports for R controller on SW#1 but the errors remain the same. During cable removal however additional errors and alarms appear as communication is fully lost from R controller but as soon as the network cable is connected and network comes back online, only the initial diagnostic alarms remain.

We have GE switches installed and we thought of replacing the switch but when ever after a power cycle through the switch everything came back online, we now believe the fault is somewhere in the R controller.
 
Hi James,

Can you please share the notes to help us troubleshoot further?
Hi MnM

Have a reed on GEH 6721 VOL1

I attach here a screenshot from the manual..( the right page to read) for everybody here....

Follow these instructions for this issue...




I wish i could be there ( at site ) also as CSA stated to support you ...

These GE stuffs are wonderfuls...

Lets give us an update on the isssue if it still persiting....

I am convinced that you will have to do some reading ( litterature in french ) on OEM manuals to get that issue solved

Have also a search on that white nutton located in bottom of UCSB
..It looks more like a backup/restore button than a reset button...

People from Nuvo pignone ( now BH ) are not very conforatble /familiar with GE mark6e Like people in Salem ...

Is that unit packaged by BH.. if this is the case ,Keep push on them to guid you in this issue..



Regards,
James
 

Attachments

Hello CSA,

I need your help regarding GE Toolbox ST software license Dongle.
Can i use license dongle that came with toolbox ST V07 software for Toolbox ST V04 software version.

Thanks in Advance.
 
Top