Dear fellow Control Enthusiast,

I woluld like to seek your expertise and assistance on hw to restore crashed HMI cimplicity Workstation

We are now struggling on restoring our crushed CRM2_SVR HMI with Cimplicity station. For brief information, this HMI can monitor both turbine operation ( 2 unit GE 7FA). WE can manipulate both control toolbox in both CRM1 and CRM2 station. Now that CRM2 crashed, how we can restore it and what important application do we need to restore back to make it work again.

Please help.

waiting for your reply.. Josh

 

We don't know how old the HMIs are at your site, nor what the OS is.

But, it would seem that there is no regular or periodic back-up procedure for the site which would have created "image" disks (CDs or DVDs) which could be used to restore a known working configuration to a new hard drive?

Back-ups are CRITICAL for GE HMIs. GE has been providing a software application, Acronis, for this purpose on HMIs in recent years. It's very easy to use, and produces excellent "recovery" disks which have saved many a site.

Recovering from a <b>crush</b> is very easy with recovery disks. The recovery disks are basically a copy of the entire contents of the hard drive with a self-install routine, so all one need do is configure the PC to boot from the CD/DVD and insert the disk, and follow the prompts and voila! you have a working HMI in about 30-60 minutes.

In the absence of such disks/method (some people actually install a second hard drive and copy the entire contents of the original hard drive to the second hard drive and save it for recovery purposes), GE usually provides a "recovery" CD inside the PC. This disk <b>only</b> reinstalls the OS and whatever GE software was originally shipped with the HMI. That means it probably only has TCI/CIMBRIDGE and maybe an early version of Toolbox/eTCSS. It probably doesn't include CIMPLICITY, and likely doesn't include any site-specific software (.m6b files, CIMPLICITY display files, CIMPLICITY Project files, etc.).

If you don't have recently created recovery disks for that HMI, I would say you should probably engage the services of GE or some other vendor to come and assist you with your endeavour. Recreating CIMPLICITY projects for multi-unit HMIs can be very complicated, especially if the site has EX2000 or EX2100 exciters and static starters (LCIs). It can be very daunting, and, unfortunately, it's not possible to just copy files from another multi-unit HMI. (Well, it's possible, but it takes a lot of manipulations and machinations and some luck (hopefully good luck!) to make it all work. Better to pay someone else to have the responsibility to make it all work again if you're not familiar with CIMPLICITY and TCI/CIMBRIDGE and GE HMIs.

And, if the site doesn't have software and a procedure for frequency, periodic back-ups of the HMIs it's a great time to put such a practice in place!

Lastly, some GE HMIs were actually configured with RAID multiple hard-drive configurations. Sometimes, depending on the severity of the crash, it's possible to recover from a single hard drive failure using the RAID function/feature. I've never done it myself, but I have heard anecdotal evidence that it has been done.

 

Josh,

I would echo all that CSA said and add these comments.
It will very much depend on how old the HMI is that you are using as to how you can restore the machine. I can tell you from my experience on machines I have that were built in 2003 running WIN2K operating system that GE provided restore disks in each machine that have an exact image of the machine when it was shipped from the vendor.

At this time GE was still using a version of Norton ghost for backup/restore tasks. I have used these disks to restore a machine with a failed hard drive with success.

Again it depends on how familiar you are with using Ghost, cimplicity, toolbox, etc. as to how things will work out for you. If you simply had a failure of the hard drive I would recommend replacing the hard drive with a new unit. This would give you the option to pull files from the old hard drive that were not part of the original backup.

If you are experiencing a failure of the motherboard then things get more difficult. As CSA said this will not be a simple task, and trying to learn if you have not had much experience with how GE arranges files, pseudodrives etc. you would probably do best to get outside help.

From there hopefully you can better prepare yourself for problems in the future. As CSA said GE is now using Acronis for backup images, this software has some incredible features, including a universal restore option that allows you to install an image on a new machine with completely different hardware in the case of a motherboard type of failure.

As always I think many people here will try and assist you in your endeavor, but some procedures are difficult to fully explain here, and we would need exact details as to machine, software, etc etc to make correct suggestions.

 

Dear CSA,

Thank you for your reply, I just now came from our day off. By the way, GE has some CDs being turn over to us during the last commissioning period. We can see the note on the CDs "Back up CRM2_SRV" on it. The thing is for 4 yrs of plant operation, so many changes has been made like I remember they upgrade the Mark VI Toolbox, the CImplicity version and others which not taking care of the new CD back up.

For the info, we are using Windows 2000 service pack 4, Control toolbox-Release 11.4, Ver V11.04.07C (Build 7574)

Cimplicity Version 5.5 (Build 3316- HMI 5.5 Service pack 3)

And you mentioned the EX 2100, yes we have it here also. Just I remember when one GE TA came here and said, when configuring 1 point like EGD point, just be careful because this will need to reconfigure to all workstation, and making mistake will make more problem.

So I believed we need to ask the GE assistance on these matter. or maybe I can try to restore the previous back up CDs,?

 

Dear MKVI

Just now we learn this back up things to do because of our workstation series of failure.

CRM1_SVR has been backed up now using the Norton ghost 2003. Probably it is our mistake not to take the back up before and every time they doing the upgrade. We now learn from our mistake which hopefully in the future will help us to attend this kind of problem for having the right back up CDs.

By the way, can you give briefly what information you need in our system so I can make it for send it to your for your reference.

 

Dear Josh,
Glad you were able to locate the CD's that have been spoken of. As mentioned they should be an exact image of the machine as it was shipped from the GE vendor, who is usually CCS Inc. If you had not done any upgrades to toolbox or Cimplicity I would have recommended using these CD's to restore the station. But since multiple things have changed this no longer sounds possible, but at least now you realize the value of having a good backup image of a machine. The most important machine to backup consistantly is CRM1_SVR since it holds all your master files.

You don't say exactly what failed on the machine. Do you know if it is a failed hard drive, or some other type of hardware failure? Quite a few computer manufactures(Dell especially) were effected by a large run of bad capacitors, you might take a good look at the motherboard for evidence of motherboard capacitor failures. We have repaired several of ours by replacing cap's only.

You really need to know what is causing the computer to crash. Is it some corrupt windows files, unreadable hard drive sectors, or a problem with the processor or hardware. The type of failure will impact how you need to try and fix it.

Without having a good image of the machine as it was running it will not be a simple fix. But with the old hard drive and a good image of CRM1 it should be possible for a proficient person to rebuild the station with minimal file loss. This is especially true if all workstations work the same way. Not much changes on the other workstations if they are configured to operate the same, and no upgrades have been performed. Usually with a good .m6b file and a recent .hmb file a station can be rebuilt fairly easily.

Take a look at your CRM1_SVR, under "Start", CDRW system and folder backup and restore. This has some nice instructions on how to backup and restore. But at this point I would not try and use the backup CD's you have since they are so old.

Write back with more information when possible.

 

Restoring of the cimplicity work station.

ok this is going to take a pretty long time. but here goes

i am uploading the manual i compiled after i commissioned my second GT. yes it is a compiled manual 90% of the pages in there are from GE standard manuals. i just rearranged it in a manner so that it will be helpful for new commissioning engineers. read through it and you will know how the software suite for the

control system is configured from the scratch. with the instructions given below and the manual you will be able to restore your cimplicity work station.

http://www.2shared.com/document/i1ic8rWE/understanding_mark6_software_u.html

1. Ensure that the network switches are working properly - GE uses allied tellyson or Cisco switches in the networks. as you already had a running system you

most probably need not configure it again. first thing you should do is to understand the network diagram. as you say that you can control the MArk VI from both

the stations using the toolbox , it is most probably in the UDH. GE uses tried and tested designs , i have drawn a sample diagram here which will resemble your configuration. you need to know the following things.

http://www.2shared.com/photo/i0IFyO3f/network_sample.html

a. IP address of the R/S/T processors.
b. IP address of the both computers on the UDH network

1.1 you said that you have a back up of the computer , please install it this will save you a lot of trouble. dont worry about the EDG and all , the main application code is already in the processor and the copy of it is in the *.m6b file you will be having in the working computer. you will just have to copy and paste it in the new computer. If you are doing a manual installation , then you well need to install at least these minimum software in the computer

a. Control system toolbox
b. EGD service and utilities
c. SDB servie , sdb utilities and SDB exchange
e. TCI sevice , TCI utilites and TCIMB
f. Data historian
g. NTP

all these software will be on the control system solution CD you will have. Install the cimplicity after you have installed the control system solutions.

using the back up or manually installing the software you should have these following drives in your computer.
C:\
E:\
F:\ pseudo drive ( created by TCI referencing E:/site , if you have a earlier version you might have to create a pseudo drive yourself)
g:\ pseudo drive ( created by TCI referencing c:/program files/control system solutions/TCI , if you have a earlier version you might have to create a pseudo drive yourself)
m:\ pseudo drive ( referencing E:/master)

ensure that you have connected the computer to the right port in the network switch.

open the command prompt and type ping (ip address of the r/s/t processor) eg ping 192.168.131.101
it will display a message, it should read no error. a sample message will be like this

ping statistics for 192.168.131.101
Packets : sent = 4 , Received = 4 lost = 0 (0% loss)
this message means that your computer is connected to the right switch and the switch is communicating properly with the processors.
if it reads something like this
Request timed out (4 times)
ping statistics for 192.168.131.101
Packets : sent = 4 , Received = 0 lost = 4 (100% loss)

this means that you are in big trouble. check the network diagram once again to see if the IP address is right , if you have connected the computer to the right switch , your net services are running (by default it will be ). This is the first stage to recover your computer. you have to get this right before proceeding further.

2. Setting up the control system toolbox , edg service , SDB and SDB utilities. - you have a working computer there , and as you have two GT's you will have

two *.m6b files in your system. find the latest version of the m6b file ( this is the one when opened will not show any major or minor difference but equal) this is the copy of the controller application configuration. copy it from the running computer and paste it in the recovered computer in the same directory as in the running one( not exactly needed but keeps things symmetrical ).
eg. copy from computer 1 E;/master/unit1/gt1.m6b
to computer 2 E:/master/unit1/gt1.m6b

take the services list from your recovered computer and ensure that the following are running

a. cimplicity HMI service
b. cimplicity viewconnection service
c. data historian
d. DHCP client
e. DNS client
f. EGD service
g. Network time protocol
i. SDB server
j. TCP/IP sevices
k. SNMP service (not exactly needed)
l. Turbine control interface
m. web client and workstation

all these services must be running , and if it is not start from the services control panel.

open the copied *.m6b files in the recovered computer. give a put to database and get from database. (sometimes it will ask for the SDB directory if that is the case then you must manually search for the SDB directory. it will be mostly in the E:/master/SDB ). give a build and validate command but do not give a download command. Now connect to the designated or any other processors through the toolbox. hope fully when you do this the status message at the bottom of the toolbox should have " 'R/s/t' READ Control Equal " written in it. take any EGD point from the hardware and io definitions and check to see if the real time value is displaying or not. take a alarm signal from the EDG and force it to see if the respective alarm is coming in the working cimplicity panel. if it comes then it means that the EGD service is running properly in the system.

another important thing is to open the *.sys file , better still you can copy it form a the working computer. you should do a build and validate. next thing is the *.hmb file which is unique for each of the computer work stations. it represents the HMI configuration of your computer workstation, your restored work station will have one too. find it and give a build and validate . this updates the SDB and other directories also. please go through the HMI configuration part in the manual i have updated to have more information.


3. setting up the cimplicity HMI - for cimplicity to work properly the SDB needs to be properly configured , the EGD service running , TCI and TCIMB running . the main project file for the cim project will be in F:/cimproject/"projectname".gef. in the restored computer you will have the same. if the above mentioned steps are correctly done , then the cimplicity project should work. the trouble shooting of the cimplicity is a tedious process and i cannot guess from here. problems generally arise in three main areas

a. wrong port/device configuration in the cimplicity
b. EGD not communicating with the cimplicity ( if you have done a EDG modification some tags may not work)
c. TCI and TCIMB not communicating with cimplicity

hopefully if the first two steps are followed with a copied cimplicity project from the working computer you will be able to have the work station ready. not being in site this is utmost what i can do from here .

 

Dear MKVI,

The problem on our CRM2 was the Hard drive itself, its not booting at all. WE tried to replace the hard drive and install new WIndows 2000 application. after installation pf Ghost 2003, we tried to recover the back up file but no avail, the restoration prompT on Ghsot was looking for file which can not find on all back up disk as we tried. We tried to install Ghost 9 but still same behaviour, it was looking for the disk which contain the particular file but even trying to all back up disk it was no avail.

WE tried to ghost copy the other site WOrk station ( for info we have 3 sites, all are having 2 7FA unit and all work station are almost identical to each other.) we ghost copy CRM2 on other site and restore on our CRM 2 work station but still in won't work. we check the address, the computer name network but still CRM2 is not communicating or even can not ping by on CRM1. Is there any other sections or configuration I need to check?

 

Dear PV,

Thank you for the very informative documents, if ever after re installation of Ghost backup and nothing is happen, i will going to try this steps you gave to us and hopefully positive result came.

Like wise, for MRVI, it seems our back up CD copy here in our site is old already like what you said, we been upgrading the toolbox and cimplicity before and so many things already happened to our work station. But hopefully some one from the group can guide us for making the Ghost copy we took on other site to work..

Thank you for all your support..

 

Dear All,

We been successfully restored the Mark VI CRM2 workstation and now working properly, Thanks to Process Variable, CSA, And MKVI for your information and help.

More power to you guys.. you help a lot.

Josh

 

Thanks for the feedback!

"Feedback is the most important contribution!"(c) here at control.com.

Don't forget to make those images for future recovery needs!!!!

 

Josh

Thanks for the feedback , good to know that you have got the workstation up and running. Ahem .. its actually process value , but process variable sounds cool too .

 

Dear PV,

I can say you contributed a lot.. again thank you and more power.

For all you guys, you really help a lot again more power and Control.Com. hopefully in the future I can share too.

Josh