K
Friends...
I am looking for advice or any possible insights any of you may have on how to proceed next with the issues outlined below.
At this Kraft Canada facility we produce bulk cheese (640 lb blocks), liquid whey concentrate, whey powders, cheese powders and ready for retail wrapped pieces of cheese (sticks, blocks, single portions, powders and shreds). I have responsibility for all Y2K remediation efforts at this and two other Kraft Canada plants.
Here is my problem...
In one section of the facility we have a batching and drying operation. In this operation, various cheese products and other food grade ingredients are mixed together to form a slurry. This slurry, after being cooked and pasteurized in a horizontal laydown cooker (that uses culinary grade injected steam) is pumped to the top of a conical shaped rotary spray dryer. In the spray dryer, the water content of the slurry is evaporated by heated air. The output of the spray is "Cheese Slurry Powder" that falls to the
bottom of the dryer as cheese "snow". Further processing then finishes the powder and prepares it for packaging.
The batching & cooking area and the spray dryer section shared a common control structure. This structure was laid out as follows... A Siemens TI
545-1104 took care of process control in the batching & cooking area. A Siemens TI 555-1102 did the same job in the spray dryer area. The 545 had only local I/O connected to it, while the 555 had both local and 4 remote I/O racks. Both PLCs were connected (by NIMs) to a Ti-Way network. The Ti-Way network was then connected to a AST 80386 PC running SCO Unix as its OS. Running on the Unix platform was the TI SCADA package called "Ti-Star". The main "Ti-Star" terminal was the HMI for the spray dryer area while a secondary "Ti-Star" terminal (on a 386 DOS box that talked RS-422 back to the Unix server) performed the HMI function for the batching & cooking area.
Along comes Y2K and we discover that the old Ti-Star application sitting on the SCO Unix box is not Y2K compliant and must be replaced. So we built a new system...
Our corporate standard called for a solution that consisted of USData's Factory Link ECS (the Y2K compliant 6.1 for Unix version) sitting on top of HP-UX (10.02... also a compliant version) running on HP9000 class machines. Ethernet was identified as the media of choice. The older 545-1104 in the batching & cooking area was replaced with a new 555-1106 due to a need for more memory.
A Cisco Catalyst 1912 switch was to act as the "hub" for all of the Ethernet connections. This switch provides us with a dedicated 100 Mb channel for the HP server and a full 10 Mb of bandwith on each of the other ports. In order to connect the PLCs to the Ethernet media, two TI 505-2572 (the CTI card) Ethernet/Serial cards were purchased for installation in the local racks.
It was decided to use NCDs Explora 700 XWindows thin clients along with "BIG" (wish I had them at home... LOL) monitors as the Operator stations.
So connected to the switch we have... The 100 Mb channel for the Server, a 10 Mb channel to the 555 on the dryer, a 10 Mb channel to the other 555, a 10 Mb channel to the dryer Op station (HUGE
XTerm) and another 10 Mb channel to the other Op station located in the batching & cooking area. Cisco diagnostics have reported a demand loading on the switch of only 3%.
The programs in both PLCs were created using TIs APT (a pre IEC 1131-3 standard language that includes SFCs, CFCs in a graphical construct, an excellent tool by the way). No changes to the base programs other than variable reading and writing in multiple blocks were required by the remediation.
The HP9000 we purchased has more resources then Carter has Liver Pills... It has a 160 Mhz RISC CPU, 256 Mb of ECC RAM, 1 Mb of L2 Cache, Twin 100BaseT ports, Twin Hot Swappable Mirrored 9 Gb SCSI drives, a 12 speed SCSI ROM drive, a 4 Gb
DAT drive and a monster of a UPS. In essence this platform is... every grown up fan of the "Commodore Vic 20" dream.
So what is the problem... The response time of the system is ridiculously slow. From the station in the batching & cooking area we see response times of 4 to 5 seconds while on the dryer portion of the system it is not un-common to have the HMI update 10 to 12 seconds after an event. With the resources that exist in the
HP9000 and the closed to other traffic Ethernet segment this control system should scream. To date we have done the following...
1) Ensured that ... the switch is in "Full Duplex" mode
2) Checked loading on the switch (its at 3%)
3) Ensured that HP-UX is configured as per our standard for optimized performance (we have about 50-60 similar systems in operation at other plants).
4) Removed some HP-UX daemons that are not truly required to be in memory.
5) Verified that communication of tag data across the system is as homogenous as possible. The EDI driver that is used (Dastec) takes block reads and writes into consideration and performs
optimization on its own as well.
6) Sent a MPS (Multi Platform Save file of the FL ECS source code to USData for analysis to ensure there are no issues with the application as built by the integrator. Who by the way have been
excellent at helping to address this issue.
Identified Possible Solutions...
1) Migrate the FL ECS code to Compiled Math and Logic from the current Interpreted Math and Logic. Means switching from PowerVB source code to compiled C++ object code that is then linked in to create the executable.
2) Migrate the FL ECS code to an exception read/write basis where if a variable changes then the impacted variable is read/written instead of the current situation where we read/write the entire tag dictionary. Means going from two routines to literally hundreds of sub routines.
At this point my prime theory is, is that communication traffic across the back plane of the 505 rack between the CPU module and the 2572 Ethernet card is restricting to throughput. I would like to move from a Ethernet EDI interface into the HP server to a Profibus one. As the Profibus port is native to the CPU module, this removes any chance of the backplane impacting
communications throughput. As this is an expensive fix to try... I am looking for comments from this august body.
I know this message was long winded... but I wanted to give you all as much data as possible. Thank you for your time and I look forward to any comments from the group. There are 2 simple
.bmp diagrams that illustrate this -- if you'd like to see them, contact me directly.
Best Regards... Rick Kelly
Project Technician
Cheese Operations
Kraft Canada
(613) 537-8069 V
(613) 537-8057 F
[email protected]
I am looking for advice or any possible insights any of you may have on how to proceed next with the issues outlined below.
At this Kraft Canada facility we produce bulk cheese (640 lb blocks), liquid whey concentrate, whey powders, cheese powders and ready for retail wrapped pieces of cheese (sticks, blocks, single portions, powders and shreds). I have responsibility for all Y2K remediation efforts at this and two other Kraft Canada plants.
Here is my problem...
In one section of the facility we have a batching and drying operation. In this operation, various cheese products and other food grade ingredients are mixed together to form a slurry. This slurry, after being cooked and pasteurized in a horizontal laydown cooker (that uses culinary grade injected steam) is pumped to the top of a conical shaped rotary spray dryer. In the spray dryer, the water content of the slurry is evaporated by heated air. The output of the spray is "Cheese Slurry Powder" that falls to the
bottom of the dryer as cheese "snow". Further processing then finishes the powder and prepares it for packaging.
The batching & cooking area and the spray dryer section shared a common control structure. This structure was laid out as follows... A Siemens TI
545-1104 took care of process control in the batching & cooking area. A Siemens TI 555-1102 did the same job in the spray dryer area. The 545 had only local I/O connected to it, while the 555 had both local and 4 remote I/O racks. Both PLCs were connected (by NIMs) to a Ti-Way network. The Ti-Way network was then connected to a AST 80386 PC running SCO Unix as its OS. Running on the Unix platform was the TI SCADA package called "Ti-Star". The main "Ti-Star" terminal was the HMI for the spray dryer area while a secondary "Ti-Star" terminal (on a 386 DOS box that talked RS-422 back to the Unix server) performed the HMI function for the batching & cooking area.
Along comes Y2K and we discover that the old Ti-Star application sitting on the SCO Unix box is not Y2K compliant and must be replaced. So we built a new system...
Our corporate standard called for a solution that consisted of USData's Factory Link ECS (the Y2K compliant 6.1 for Unix version) sitting on top of HP-UX (10.02... also a compliant version) running on HP9000 class machines. Ethernet was identified as the media of choice. The older 545-1104 in the batching & cooking area was replaced with a new 555-1106 due to a need for more memory.
A Cisco Catalyst 1912 switch was to act as the "hub" for all of the Ethernet connections. This switch provides us with a dedicated 100 Mb channel for the HP server and a full 10 Mb of bandwith on each of the other ports. In order to connect the PLCs to the Ethernet media, two TI 505-2572 (the CTI card) Ethernet/Serial cards were purchased for installation in the local racks.
It was decided to use NCDs Explora 700 XWindows thin clients along with "BIG" (wish I had them at home... LOL) monitors as the Operator stations.
So connected to the switch we have... The 100 Mb channel for the Server, a 10 Mb channel to the 555 on the dryer, a 10 Mb channel to the other 555, a 10 Mb channel to the dryer Op station (HUGE
XTerm) and another 10 Mb channel to the other Op station located in the batching & cooking area. Cisco diagnostics have reported a demand loading on the switch of only 3%.
The programs in both PLCs were created using TIs APT (a pre IEC 1131-3 standard language that includes SFCs, CFCs in a graphical construct, an excellent tool by the way). No changes to the base programs other than variable reading and writing in multiple blocks were required by the remediation.
The HP9000 we purchased has more resources then Carter has Liver Pills... It has a 160 Mhz RISC CPU, 256 Mb of ECC RAM, 1 Mb of L2 Cache, Twin 100BaseT ports, Twin Hot Swappable Mirrored 9 Gb SCSI drives, a 12 speed SCSI ROM drive, a 4 Gb
DAT drive and a monster of a UPS. In essence this platform is... every grown up fan of the "Commodore Vic 20" dream.
So what is the problem... The response time of the system is ridiculously slow. From the station in the batching & cooking area we see response times of 4 to 5 seconds while on the dryer portion of the system it is not un-common to have the HMI update 10 to 12 seconds after an event. With the resources that exist in the
HP9000 and the closed to other traffic Ethernet segment this control system should scream. To date we have done the following...
1) Ensured that ... the switch is in "Full Duplex" mode
2) Checked loading on the switch (its at 3%)
3) Ensured that HP-UX is configured as per our standard for optimized performance (we have about 50-60 similar systems in operation at other plants).
4) Removed some HP-UX daemons that are not truly required to be in memory.
5) Verified that communication of tag data across the system is as homogenous as possible. The EDI driver that is used (Dastec) takes block reads and writes into consideration and performs
optimization on its own as well.
6) Sent a MPS (Multi Platform Save file of the FL ECS source code to USData for analysis to ensure there are no issues with the application as built by the integrator. Who by the way have been
excellent at helping to address this issue.
Identified Possible Solutions...
1) Migrate the FL ECS code to Compiled Math and Logic from the current Interpreted Math and Logic. Means switching from PowerVB source code to compiled C++ object code that is then linked in to create the executable.
2) Migrate the FL ECS code to an exception read/write basis where if a variable changes then the impacted variable is read/written instead of the current situation where we read/write the entire tag dictionary. Means going from two routines to literally hundreds of sub routines.
At this point my prime theory is, is that communication traffic across the back plane of the 505 rack between the CPU module and the 2572 Ethernet card is restricting to throughput. I would like to move from a Ethernet EDI interface into the HP server to a Profibus one. As the Profibus port is native to the CPU module, this removes any chance of the backplane impacting
communications throughput. As this is an expensive fix to try... I am looking for comments from this august body.
I know this message was long winded... but I wanted to give you all as much data as possible. Thank you for your time and I look forward to any comments from the group. There are 2 simple
.bmp diagrams that illustrate this -- if you'd like to see them, contact me directly.
Best Regards... Rick Kelly
Project Technician
Cheese Operations
Kraft Canada
(613) 537-8069 V
(613) 537-8057 F
[email protected]