This is the last gasp migration for Microsoft ecosystem 16-bit applications. Windows Server 2008 x86 is the last Microsoft server operating system to support them. You can upgrade from Server 2003 to Server 2008 and buy yourself a few more years, but extended support for Server 2008 runs out in 2020.
The migration won't be smooth. There are a whole bunch of "little things" that don't quite work well with 16-bit applications under Server 2008. Printing is one. Especially under Remote Desktop Services (formerly known as Terminal Services). There are workarounds for pretty much everything, but be prepared to spend a lot of time on Google trying to find them.
Remember that just because your application appears to be a 32-bit application with a Win32 GUI and all the window dressing of a 32-bit Windows application doesn't mean it is. Lots of 32-bit applications call 16-bit components.
If you have any reason to doubt that your seemingly 32-bit application might not be entirely 32-bit there is a simple way to check. Install the application on a clean, unpatched Server 2008 x64. It will either work or it won't.
The chances of a true 32-bit application which works on a fully patched Server 2003 R2 SP2 system not working on a clean, unpatched Server 2008 x64 are slim to none. You might get a bark about dependencies, but that's not the same as it "not working". Install the relevant additional applications and try again.
If your "32-bit" application still refuses to work on an x64 copy of Server 2008 then the chances are that it's calling some piece of 16-bit code somewhere. Start asking the developers pointed questions.
The incremental approach to upgrading under duress
There are some practical steps to be taken in dealing with 16-bit applications. For the most part, these steps also apply to non 16-bit applications, but for reasons we'll discuss later, you're very unlikely to need to worry as much about those.
The easiest way to figure out how to handle them is by using virtualisation. You can snapshot VMs after every change, making rolling back to a previous state very easy. This is wonderful for testing.
If you can't use virtualisation – for example, your application needs to talk directly to a hardware controller because it drives some important machinery – then it's time to invest in some imaging tools. Symantec Ghost is the most famous example.
If you are migrating 16-bit applications to a newer version of Windows you need to start by installing Server 2008 x86. Do not update it. Install your application and port over any relevant data, configurations, etc. If you can get help from your developer in identifying how to copy your data and configurations from one system to another, great! If not, be prepared for a lot of trial and error. Document everything.
Once you have your application up and running on your Server 2008 x86 installation, double check your documentation to make 100% sure you know how to migrate things. Now, reformat the server and perform the migration again, this time without all the trial and error.
At this point, you should be old hat at moving the data and configuration for your application from one server to another. You should also have a clean, working install on a non-updated server. Take a snapshot of this server (if it is in a virtual machine) or a ghost image (if it is on metal). You'll need this "known good" configuration later.
Next, perform all security updates for Server 2008. No optional updates. No non-security "critical" updates. No internet explorer updates. Just the security updates.
Test your application thoroughly. If it no longer works then reload the known good (clean) image and install the security updates in batches. Install them in blocks of 10, starting at lower numbered KB patches (older) and working your way towards the newer ones. Test the application thoroughly after each block.
Once you find a block that breaks things, uninstall the last batch of ten patches and install them one at a time until you find the one that breaks your application. Write the KB down for later and don't install that patch. Continue patching in blocks of 10 and testing after each block. Every time you find a patch that breaks things, write it down, don't install that patch and keep going.
Continue on through all remaining "critical" patches that are not security updates and are not Internet Explorer. After everything else is said and done, try installing Internet Explorer updates. Internet Explorer patches are the most likely to break everything horribly, so if nothing has gone wrong until now, you aren't out of the woods yet!
If you have made it through all patches without things breaking, congratulations! If not, it's time to sit down with Google and the KB numbers of your patches. Let me be clear here: use Google. Don't use Bing. If you try to use Bing to solve patch issues for Windows Server using KB numbers you are going to go mad. Under no circumstances use the search features on any Microsoft website. They're somehow actually worse than Bing.
If you can't find the solution, call the developer (if they still exist). If the developer can't (or won't) help you, try asking on the Spiceworks forums and/or Microsoft's Technet forums. In almost every case not related to printing, a hotfix will exist, it's just going to be a pain to find it. If all else fails, call Microsoft support, but be aware you might be charged a pretty penny.
Once you've figure out the workarounds for any patches that are causing you problems I strongly recommend resetting your system to your original "known good" install. Rerun your patches carefully, applying whatever hotfixes or workarounds are required in the proper order. This will generate a new "known good" system where you have not uninstalled any patches or otherwise done any trial and error.
Take a new master image of your system. Make several copies and make sure at least one copy lives offsite. Also make sure a copy of your documentation for a clear rebuild lives offsite as well, just in case. Make sure that other people in your organisation know where that offsite location is and have access to it, just in case you get hit by a bus.
The "how to build from ground zero" is especially important if you are installing this all on raw hardware as opposed to virtual machines; backup images don't tend to clone well from one set of hardware to another. If you're still using 16 bit applications to talk to hardware then the chances are the hardware you're using is a little on the "hard to find" side, and replacement parts might not be like for like. Your job may one day rest on your ability to reinstall this all from scratch, and thus on the documentation you make during this migration.
Server 2008 is quite old. It's your only choice for those 16-bit applications, but if you have to use it, install third party security software. Additional layers of security are important. Anti-malware, a better firewall and an intrusion detection system are all good plans. After you're done, you guessed it: test, document, image.
32-bit applications on death row
16-bit Microsoft ecosystem applications have until 2020. It is time to plan their migration to a 64-bit successor. 32-bit applications aren't a safe haven, however, and anyone still using them needs to start planning an exit as well.
Microsoft recently unveiled plans for nano, the future of Windows Server. Nano does not support 32-bit applications. The writing on the wall is clear: 64-bit only is the future, and the number of refresh cycles remaining until the runway for 32-bit applications runs out is numbered.
Beware Internet Explorer
My biggest upgrade woes come from Internet Explorer. Microsoft broke involving printing between Internet Explorer 7 and Internet Explorer 8 and they refuse to even acknowledge the existence of the problem, let alone do anything to fix it.
On a Server 2003 or Windows XP box running Internet Explorer 7 applications that are designed to call Internet Explorer in order to print HTML files work fine. Upgrade to Internet Explorer 8 and they suddenly start working intermittently.
Every Internet Explorer version since has the same problem: if you have an application whose job it is to sit there and print HTML files it will print just fine for a random number of files before dying. For reasons unknown the print spooler will (sort of) blow up, but not crash (which would allow Windows to detect a service failure and restart the service.)
Applications that can be recoded to use Google's Chrome or Mozilla's Firefox to render and print the HTML don't have this problem, nor do Server 2008 installs where Internet Explorer is deliberately prevented from updating.
It's a quirk. A bizarreness of Microsoft's ongoing updates that serves as an example of what to watch out for when migrating. It's also why you should always start from the bare, unpatched version of an OS and work your way up. You'll never know what might have happened over the lifespan of the OS that could have "broken" an otherwise usable environment...or whether or not there exist workarounds or fixes for these sorts of esoteric bugs until you perform methodical testing.
Test. Document. Image. Good luck. ®