P
Few days back we have had an incident due to which all console stations went blank and restored after 12 mins. Interestingly no changeover of Experion servers occurred and Consoles did not picked up data from Controllers directly as well.
By looking at the events generated and Switch logs we assume following had happened;
1)Multiple commands were generated to retrieve customize report by the board-man (16 each) in number from console station. This report gathers data form Server and displays it on screen of client station.
2)Soon after the commands were generated the console station (client) itself went into fault and "station failure" alarm was generated due to excessive requests of Report.
3)When directly connected switch (SW1A/B)sensed its one node is not responding it generated "SNMP TRAP Link Down" to indicate one node on the network is not available or network is going down.
4)Right after this OPC integrator failure alarm also appeared. We have been using OPC to transfer quick builder points to control builder.
5)In the meanwhile console station's (client) connectivity went bad with the primary Server one by one as apparent in Server 'Events'.
6)CPU overloading alarm of Server1 also appeared as apparent in the attached event list indicating "CPU time is greater than 90%".
It took 12 mins for indications to be restored while Server changover did not happen & Clients also did not picked up data from C300 controllers directly.
Action taken:
Switch 1AB logs were retrieved & it does not indicate any blockage of port. We assume Trap request was generated to notify the manager (Server) that one node on the link was down.
Queries / Concerns
1)Please confirm if the above scenario was resulted due to network congestion or CPU overload?
2)According to our IS department, servers are running out of RAM and need to be upgraded but on the contrary Honeywell insisted earlier that Servers with same specs have been installed and working fine even in larger applications than ours in other industries.
3)Also IS department says that Windows 2003 does not support more than 4GB RAM? if we want to extend the RAM we may be changing Windows version as well ?
4)Is it possible to increase Paging Memory to compensate above RAM requirement and relocate it to other drive on which Server application is not running? I have seen Paging file size can be increased in System & it is better to relocate it to other drive for isolation from running server application.
5)If certain PMs are required for Servers that we may follow to eliminate such discrepancies?
6)Why server remained primary? If it was in service than why no data was being transferred?
7)Why did console did not extracted data from Controllers directly and remained blanked for 12 mins?
Would highly appreciate your response on this.
Thanks & regards
By looking at the events generated and Switch logs we assume following had happened;
1)Multiple commands were generated to retrieve customize report by the board-man (16 each) in number from console station. This report gathers data form Server and displays it on screen of client station.
2)Soon after the commands were generated the console station (client) itself went into fault and "station failure" alarm was generated due to excessive requests of Report.
3)When directly connected switch (SW1A/B)sensed its one node is not responding it generated "SNMP TRAP Link Down" to indicate one node on the network is not available or network is going down.
4)Right after this OPC integrator failure alarm also appeared. We have been using OPC to transfer quick builder points to control builder.
5)In the meanwhile console station's (client) connectivity went bad with the primary Server one by one as apparent in Server 'Events'.
6)CPU overloading alarm of Server1 also appeared as apparent in the attached event list indicating "CPU time is greater than 90%".
It took 12 mins for indications to be restored while Server changover did not happen & Clients also did not picked up data from C300 controllers directly.
Action taken:
Switch 1AB logs were retrieved & it does not indicate any blockage of port. We assume Trap request was generated to notify the manager (Server) that one node on the link was down.
Queries / Concerns
1)Please confirm if the above scenario was resulted due to network congestion or CPU overload?
2)According to our IS department, servers are running out of RAM and need to be upgraded but on the contrary Honeywell insisted earlier that Servers with same specs have been installed and working fine even in larger applications than ours in other industries.
3)Also IS department says that Windows 2003 does not support more than 4GB RAM? if we want to extend the RAM we may be changing Windows version as well ?
4)Is it possible to increase Paging Memory to compensate above RAM requirement and relocate it to other drive on which Server application is not running? I have seen Paging file size can be increased in System & it is better to relocate it to other drive for isolation from running server application.
5)If certain PMs are required for Servers that we may follow to eliminate such discrepancies?
6)Why server remained primary? If it was in service than why no data was being transferred?
7)Why did console did not extracted data from Controllers directly and remained blanked for 12 mins?
Would highly appreciate your response on this.
Thanks & regards
