Upcoming Certification tests

It is that time of year again and I have decided to take a couple of certification tests to enhance my marketability and skills. Solarwinds now has the Solarwinds Certified Professional certification and I have used/recommended their products for years now. Their network management and alerting software Orion is the best in the business and I am glad to see that they have begun to offer this certification. While the test is not geared directly towards Orion and the concepts can be used with any network management software, there is a section geared directly towards Orion. I have looked through the study guide and it appears to be fairly straight forward for someone who has used the product plus I got the added bonus of receiving a free exam voucher for being one of the first 500 people to sign up for the test. This deal is running until the end of December or when 500 people register so definitely sign up if you are thinking about taking the exam.

The second certification I will be going for should be cake - it is Apple's new Snow Leopard test for the Apple Certified Support Technician. I have not taken an Apple test before but after two years of heavy use and support of other Mac's I feel that now is the time. Apple encourages you to take the test by advertising your certification on their site and allowing people to find certified support personnel in their area. This is a great idea and one that I would like to take advantage of in order to gain additional clients.

I will keep everyone up to date as to my progress and I hope to have some updates in the upcoming weeks.

Using ESEUTIL to diagnose/repair an Exchange 2007 database

Eseutil: Exchange 2007 Help

Anyone who has dealt with Exchange knows that ESEUTIL is the tool you can use to do any type of repair or integrity check on your database. With Exchange 2007 this utility has been expanded to include repairing the new queue database for your Hub or Edge Transport servers as well.

Using ESEUTIL still requires that the database be offline so that you can do any type of activity to the database such as defragging to gain the free space back from an EDB file. Normal online maintenance can run while the database is up and functional but you cannot use those tools manually on your own. Normal defragmentation can take up to two weeks to occur in an online state so if you need the additional space immediately you will have to take the database offline in order to use ESEUTIL.

The major difference in the past is now the queues are in a database format and are no longer individual flat files that are kept in various folders on your server. This can be a good or bad thing depending on how you look at it. It is more difficult to get an idea if your queue is hosed or what the offending message may be so you will need to use the queue viewer in your toolbox of the Exchange Management Console or a PowerShell cmdlet in order to figure out what is going on. This database as I have recently learned can get corrupted so it is important to know that you may have to use ESEUTIL against this database as well - it is called mail.que by default on your box.

MxToolbox - test your external email servers

Email Server Test - Online SMTP diagnostics tool - MxToolbox

Do you want to know if you have properly configured your external mail server to not be an open relay for spambots? Do you want to see if your mail server is blacklisted on various spam sites? MxToolbox is a free site that allows you to test your mail server for a variety of issues and has a simple interface to take steps to fix any problems that might be found. There are many sites that want you to pay a fee for these tests but I have found MxToolbox to be great and highly accurate.

Yesterday I ran into a problem where an external mail server got blacklisted due to a spambot infection on a machine on our network that was NAT'ing through the same IP address of our mail server. I had the machine pulled off the network within a few minutes of seeing the problem but at that point the damage was already done to our reputation and we started getting NDR's from various servers due to our blacklisting. MxToolbox allowed me to see who had blacklisted us and start the removal process all from one spot. This saved a lot of time and effort on our part instead of having to guess which list was blocking us.

Another good tip from yesterday is to block SMTP traffic flowing from your network except from your Edge Transport server. This will stop any spambot from originating on your network and not cause these types of issues. I was able to see who the culprit was by looking at all active connections on our firewall to port 25 of an outside address. I now have a firewall rule set up to block these types of connections in the future.

Differences between Standard and Enterprise editions of Exchange 2007

Microsoft Exchange Server: Exchange Server 2007 Editions

One of the questions people often ask when figuring out what version of Exchange that they actually need is the limit on the size of the database. Microsoft has done a good job of putting together a quick and simple explanation of the differences on the page above which is really helpful for folks trying to plan and manage their Exchange environment. The cost of an Enterprise license is fairly more expensive than a standard edition, but you lose the flexibility and redundancy in your organization. For example, you cannot use CCR for clustering your Exchange environment and you are limited to just 5 storage groups with 5 different databases. The theoretical limit on exchange database size is 16tb which is the same for each version of Exchange but in reality the limit is much lower.

Microsoft has a support document that states the recommended maximum size for a database is 100gb due to the I/O involved with transaction logging and backup/restore functionality. You of course can go much higher but be prepared for some headaches along the way such as log file replaying and recovery time being a lot longer. It is a good practice to try to keep the databases around 100gb if at all possible. This is another advantage of the enterprise edition in that you can manage 50 different databases on a single server instance.

Always remember to keep your logs separated from your database on different logical volumes so that the I/O can be properly spread between the two sets of data. Most of this is common sense and very logical, but if you take these steps up front you will save a lot of headache down the road.

Upgrading Exchange 2007 to Service Pack 2

This past weekend I wanted to get our Exchange environment up to date and I decided to install SP2 as well as Rollup 1 on our three Exchange 2007 servers. This article will give you some hints to help you along the migration path that you may want to consider.

Install SP2 and Rollup 1 on your Edge Transport or Client Access Server FIRST. This will allow you to continue to proxy requests to your back-end servers since SP2 is backwards compatible with SP1. If you decide to install SP2 on one of your Mailbox servers first, and your OWA proxy is not up to date, then your users will no longer be able to access their mail via webmail - they will get an error about the version level being unsupported on the back end. Amazingly though ActiveSync will continue to function in this scenario.

You do not have to reboot in order to install SP2 or Rollup 1 but do allow for some maintenance for this to occur. In my environment the install took approximately 1 hour to complete per server.

Once you have installed SP2, any changes you made to the OWA access such as SSL requirements are replaced with the defaults. I found this the most frustrating since I took some time to get this all set up the way I wanted. I wish they would leave these settings alone when you install the update. As always, document your configuration with notes and/or screenshots so that you do not get in a bind and have to re-invent the wheel once you install the update.

Overall, the process went fairly smooth and was pretty straight forward. Keep in mind that any upgrade or changes you make to your email environment should be considered critical and that backups before and after the changes are crucial. Testing extensively should be done in order to make sure that the transition will be as seamless as possible.

Using MFCMAPI to find bad messages in Exchange 2007

MFCMAPI

Yesterday I had a major Exchange 2007 outage on a server that I will speak more about later but I found a neat tool yesterday to give you some insight into what is going on with messages in your queue. Believe it or not, but sometimes a bad message or a bad meeting request can cause your Mail Submission or Transport Service to take a major headache and slow processing down to a crawl. In order to properly diagnose what is going on you can always use the Queue Manager to see messages that you can suspend processing, but unfortunately you cannot delete an individual message or request this way. Using MFCMAPI you can actually use a MAPI connection to Exchange and find and delete offending messages that you think may have been the problem. It is a handy tool to keep in your toolbox as you make your way through Exchange issues.

BackupExec 12.5 sp3 and Exchange 2007 GRT MAPI errors

BackupExec 12.5 has a technology for Exchange that they call GRT which allows for the granular backup and recovery of Exchange 2007 so that you do not have to recover an entire database in order to bring back a certain mailbox or message that was lost. In theory, this works great but unfortunately theory and reality are two different things.

I did an upgrade of BackupExec 12.5 to SP3 and before the upgrade GRT appeared to be working fine. However, after closer inspection, it wasn't really working but was not throwing errors either. The problem was that GRT uses a MAPI connection to your backup once it is on tape or disk to create the catalog of all of the mailboxes and the messages that are contained inside of them. If the account does not have full Exchange 2007 permissions on the organization or server, then this catalog process will fail with a MAPI credentials error. My original problem was that GRT was only working for a few mailboxes that actually had permissions to be logged in to from my backup account but for the vast majority it was failing. However, BackupExec relayed that the job was a success even though most of my mailboxes were failing to be indexed properly. This is obviously not good.

Once I upgraded to SP3, the problem got worse. Now, I was getting a MAPI error once the database and logs had been backed up and no indexing was taking place. I checked all of my permissions, created a new AD account with the appropriate permissions to exchange, and yet it still failed. Finally, I stumbled across an article that mentioned hardware compression as a possible culprit of the MAPI error since it was trying to log in to the image on the tape - why it does this is beyond me. I turned off hardware compression for this particular job, re-ran it and magically all of my indexes were created properly using MAPI and GRT.

I believe this to be a major bug in that you really need to verify all of the mailboxes are being indexed properly and Symantec needs to fix BackupExec so that it doesn't read the MAPI information from the backup media. This should be done against the actual database on disk in my opinion. Hopefully this will be fixed in a future release but for now, this should save you a lot of time that I went through this past weekend to solve this issue.

MacUpdate Desktop 5.0

MacUpdate: MacUpdate Desktop 5.0

One of the biggest problems with Windows is keeping all of your applications up to date and fully patched, even with enterprise patch management software such as BigFix. There are so many applications and so many places to go look for updates that it can be time consuming and a pain just to make sure everything is okay and working properly.

With a Mac, you no longer have to search and search for product updates because of a repository tool like MacUpdate Desktop. This tool will scour your machine for every app you have installed and then check its database for updates or patches that are available to you. You can then download and install all of the updates or individual ones if you so desire. Installation of the patches are seamless and do not require to be an IT person in order to make it work.

For $20 a year, this is a great tool that will keep your apps up to date and protected from any security vulnerabilities that may be out there. The database is constantly being added to so if your favorite app is not supported it will be soon. Currently there are over 30,000 apps in the database and I highly recommend this app to any Mac user.

Increase the number of telnet sessions allowed

Increase the number of telnet sessions allowed

Again a default setting that I do not understand. An application that I support uses telnet for remote access (don't ask!) and by default the number of instances configured to be allowed to the box is 50. This is a fully patched and up to date Redhat 5.4 system.

In order to change this to something a bit more sane, you need to edit /etc/xinetd.conf and change:

instances = unlimited

You then need to restart xinetd by doing:

/etc/init.d/xinetd restart

SNMPD syslog entries

Red Hat Knowledgebase: How can I fix the excessive logging of snmpd in Red Hat Enterprise Linux 4 update 5?

When you install Redhat Linux snmpd logging is cranked way up for syslog which can start to fill up your /var/log/messages file fairly quickly if you are using some sort of network monitoring software such as Orion. I personally don't want to see all of the snmpd connections coming to my server so you need to tell the snmpd daemon not to log extensively to syslog.

In /etc/init.d/snmpd, you need to change the options to be:

OPTIONS="-Lf /dev/null -p /var/run/snmpd.pid -a"

I am not sure why this is enabled by default on the latest net-snmp package but nevertheless this will fix the issue.

Adding Swap Space to a Linux server

Adding Swap Space

Obviously swap space is bad and should be used as a last resort to physical memory. However, there will be times when you will want to add swap space to a server that needs an additional area or an increased size due to application requirements.

The link above is a good overview of increasing the size of a logical volume that happens to be formatted as swap. I used this recently to increase the size of swap for a server from 20gb to 60gb on a Redhat 5.4 64bit server.