Oops: an upgrade from Trend Micro CSM 3.5 to 3.6 goes bang
One of my goals for this week was to upgrade an installation of Trend Micro's Client Server Messaging Security Suite from v3.5 to 3.6. Being a minor points upgrade, with no big changes under the hood, I was confident that it would be another smooth upgrade. Feedback from other Trend aficionado seemed to indicate that the upgrade was unremarkable with no issues being reported, and I am also hopeful that the upgrade will fix the irritating authentication errors affecting Trend's .notaccount.
Silly me - I forgot that computers are designed to keep us humble - the minute we get too confident in our own abilities they will do something to bring us right back down to earth.
Despite having a comprehensive disaster recovery protocol in place for the DC and terminal server that can get things up and running again in a very short space of time, I still feel a cold shiver when something goes wrong on a DC - downtime is a bad thing when it affects the server off which all else hangs - if minutes of downtime extends into hours or even days the financial loss to a business can be crippling. As much as I love SBS, it can be a real disadvantage to have an entire business infrastructure dependent on the one everything-but-the-kitchen-sink server if something bad happens to that server.
The first visible issue encountered during the upgrade was a fatal error during installation of the Messaging Security Agent on the DC. Before the fatal error, "uninstalling SMTP hook" had been on screen for roughly 10 minutes. One irritating thing about the Trend installer at this stage of the proceedings is that there is no cancel button - you're committed to the install and stuck waiting for it to succeed or fail, with no way out apart from forcing the install to halt via Task Manager.
Apart from the visual errors, there were also things going wrong in the background. It looks like the installer was not able to shut down Trend's running services cleanly during the upgrade. I note the following Trend related error occurred at the time of the upgrade (only noted in the error logs - nothing appeared on screen) "Faulting application PccNTMon.exe, version 7.6.0.1095, faulting module PccNTMon.exe, version 7.6.0.1095, fault address 0x00012513".
The upgrade notes do not recommend that Trend related services be stopped manually before an upgrade; in fact I have seen upgrades fail if services are not running when an installer expects them to be running, but considering what I saw yesterday it is worth doing a little experimenting to do to see what happens if Trend's services are stopped before an upgrade, because it looks like the installer is not coping well if it hits a difficulty when managing a service.
Anyway, the failed installation of the Messaging Security Agent left things in a bit of a mess. Not only was the Messaging Security Agent not installed on the DC, but all email flow had stopped, including internal mail.
The following steps were required to get the Messaging Security Agent installed and running.
- Open Add/Remove Programs
- Uninstall "Trend Micro End User Quarantine" (note that the Messaging Security Agent was not listed in add/remove programmes, having failed to install).
- Start Trend Micro Security Dashboard.
- An attempt to uninstall DC from the console failed. Therefore I had to simply remove DC (Security Settings Tab) and then add it back, installing the MSA.
I then had to re-do all of the custom settings including attachment directories, spam filtering, attachment filtering, content filtering settings etc.
We tracked down the cause of the stop in email flow which, thankfully, was not as a result of a major breakage. The default SMTP Virtual Server was not running which is quite likely related to the delay I saw when Trend was removing its SMTP hook. Thankfully, all was that needed was to start the Default SMTP Virtual Server via Server Management.
The next problem to tackle was a failure when the client was auto-updating on some desktop PCs after the server upgrade - on my network 3 out of 25 machines have so far been found to be affected by the failure (with another 4 yet to log on and upgrade) which I consider to be a barely acceptable strike-out rate.
Symptoms: Windows XP Security Centre Red shield alert warning of no antivirus on the machine. No entry in add remove programs. No Trend processes running.
Attempts to install the new client via %servername%\ofcscan\autopcc.exe failed - the CMD window appeared, then nothing. Attempts to install the client via the log-in page for the Trend Micro Security Dashboard ("Click here to start installing the Client/Server Security Agent to your computer") also failed with the error "Agent already installed".
Fix: Manual removal of what was left of the client from the desktop PCs using the instructions at http://esupport.trendmicro.com/support.viewxml.do?ContentID=EN-127417
I've only had a look at one of the affected PCs so far - the only one that is used to access the Internet or email - listed below is what I found; the rest of the PCs will be checked on Monday morning.
Step 1 of KB: missing services - Trend Micro Client/Server Agent Listener; Trend Micro Client/Server Agent RealTime scan. Trend Micro Client/Server Agent Personal Firewall service listed but not running.
Step 3 of KB: Programs entry did not exist
Step 5 of KB: All keys existed
Step 6 of KB: Key did not exist
Step 7 of KB: Key did not exist
Step 8 of KB: Only ofcpfwsvc key existed
Step 12 of KB: No devices existed.
Step 14 of KB: Folder and contents existed.
Running %servername%\ofcscan\autopcc.exe now completed successfully.
Restarting Default SMTP Virtual Server using Server Management
