Migration to XE3: Almost there

Two months back, when I was replacing all of data layer components in Delphi 6, in preparation for the move to Delphi XE3, I never imagined that part would actually be the most time consuming. I am writing today having fully completed the software migration over to XE3. (There are still some utilities and bespoke projects to be done but these are small projects that take little time and wouldn't affect the launch of a new version of our standard package).

We have still to entirely functionally test the software for any run-time issues and unexpected behaviour changes, but our entire software codebase is now ported to XE3 and running well. The bottom line is this: it was much easier than I thought it would be.

As I mentioned in my previous post on the migration to ADO, the longest part of this process has been writing out the old components and replacing them with the dbGo ones. Obviously there is a good case here for better data abstraction, but time was limited and we had to get the job done so we did it the manual and long way. Maybe that's something we can work on over the coming years but for now, the job is done and the results are good.

Most of the run-time issues we've seen are actually database-related, so nothing to do with XE3, but more to do with subtle differences in the way ADO and ODBC work, as well as syntactical differences in the components.

As for the XE3 compilation, the bulk of what we encountered was to do with the Unicode breaking changes from Delphi 2009. This was largely covered in several white papers by Marco Cantu, Cary Jensen, Bob Swart, etc when Delphi 2009 was released. My colleague and I read the papers and agreed on the recommended approach that we move to full Unicode support, rather than replace String with AnsiString, PChar with PAnsiChar, etc.

Although our software is almost exclusively UK based (because of its nature), and the fact that the data we have to submit to HMRC (the UK government tax agency) is only allowed to contain ASCII characters, we found that it was easier for us to move forward with the native Delphi types than to try and maintain the older Ansi types.

The only exception to this, where I really had to put my brain into gear, was for some API headers that I had written in Delphi for some older hardware DLLs. The hardware communication stopped working with Unicode types, so I switched those back to Ansi types and everything worked as it did before. Also, we have some units that implement some well known encryption routines (SHA-1, Base64, MD5), and these all needed attention to prevent them from using Unicode.

Here is a summary of the things we came across most often:

• Implicit string conversion warnings.
  90% of the time this was simply because different variable types were being used. In almost all cases for me personally, it was the use of the ShortString type that raised this warning, and so I made the decision to just do away with ShortString all together and always use String.
• WideChar reduced to byte char in set expressions. Consider using 'CharInSet' function in 'SysUtils' unit.
  Another compiler warning well reported in the Unicode conversion white papers, this is simply as case of changing the syntax of testing for a Char in a set of Chars.
• Syntax error using local settings constants.
  Having made much use of constants such as ShortDateFormat, ShortDayNames, etc, these now have to be prefix with the FormatSettings global variable.
• [dcc32 Hint] Inline function 'TFont.GetStyle' has not been expanded because unit 'System.UITypes' is not specified in USES list.
  The name of the class and method often changes, but this compiler hint is always resolved by following the instruction: add UITypes to the unit's Uses clause.
• Version Info and Release v Debug.
  This one really caught us out and I wasn't aware of it until we switched to XE3. Firstly, projects now have separate Release and Debug build configurations, which means any project options you have need to be set individually.
  
  Not a problem for the most part, but the Delphi 6 project file's (.dpr) VersionInfo doesn't migrate across correctly into XE3. The Language Locale reverted back to English (United States), where we had previously had it set to English (United Kingdom). Also, the information only came across for the Release configuration, so needed to be redone in the Debug configuration. We use the information in here for checking which modules can be loaded at run-time so it was important that it was set correctly in both configurations.
  
  Personally, I don't like having to switch to Debug mode into order to be able to set breakpoints and step through code, and then have to remember to set it back to Release mode for the final build. I guess you have the option to set both configurations to use the same Compiler settings, but this would add unncessary debug information to the compiled release executable, so I can see the point of having 2 options. I suppose it is just a case of having to get used to a new routine.

There were some nice new features that we were able to make use of and throw out some more third-party components and units, which I wasn't expecting. The native TZipFile type meant we could support Zip files without relying on a third-party unit or tool. And the RegularExpressions unit meant I could also get rid of an old unit written for Delphi 6 that enabled it to handle regular expressions.

I was also able to get rid of the manual manifest files I had created to allow Windows theme support, as this has been a project option for a long time in Delphi, but obviously not way back in D6.

I'd like to talk about the IDE and performance changes but I will save these for another post. But for now, I'm delighted with how well the migration has gone and the results we have. If you're thinking about doing it, but had the same reservations I me, then I can now say from experience that it isn't as bad as you think and it's worth the effort.

Migration to XE3: ADO complete

If you've been following this (often tortuous) path of how we are going to migrate our 14 year old software from Delphi 6 to XE3, you'll know that our only real stumbling block was the fact we were using the now defunct ODBCExpress database component set. We needed to move to a different tool that was supported by XE3, and so we finally decided upon the dbGo ADO components that have been native with Delphi since Delphi 5.

It is with great satisfaction that I can report we have now moved all of the standard code over to the ADO components. There are still some bespoke utilities that we will come to later, but for now the standard software is up and running using ADO. It took us a little over two weeks of solid work to code out the old component set, and this was all done in Delphi 6 so that we could get ourselves to a point where, in theory, everything was working the same as before.

I'm delighted with the speed improvement we've seen in database access. I'm not sure whether this is because ADO is an improvement over ODBC, or the drivers are better, or whether the ADO components are more effective than the ODBCExpress components were. Probably a bit of everything.

It's kind of funny timing that just as we finished this refactoring, Embarcadero announced the purchase of AnyDac (now called FireDac). A few years ago when I first blogged about needing to move away from ODBCExpress, a few correspondents recommended AnyDac. I've heard nothing but good things about it, but ultimately I wanted to get away from a non-native third-party component set which is why we opted for dbGo. I'm pleased with this decision still, but I certainly would have given AnyDac a closer look had I known that it would become standard issue by the time we got to XE3.

I'd like to take the opportunity to thank my colleague Andrew for his collaboration on this task. I've been putting this off for a while now but having another pair of (safe) hands has given us the confidence to go for it, and Andrew has effortlessly ploughed through the same tedium as I have in order to get us to where we need to go.

So now, the next step is to open the code in XE3 and work through the Unicode warnings. I'll keep you posted.

Delphi 6 to XE3: SOAP and IP*Works SSL

Although we have not officially begun the migration phase yet from Delphi 6 to XE3, I have started installing our third-party component packs to ensure they are all working.

One of our products requires communication with the HMRC, the UK's tax man (equivalent of the IRS in the United States). This communication is done using SOAP, using a secure address. Originally, I had planned on writing this using the native SOAP components introduced in Delphi 6. However, the specification from HMRC showed that the SOAP envelope had to contain a SOAP header.

The HTTPRIO in D6 did not have a SOAPHeaders property, and so we made the decision (rightly or wrongly) to buy the IP*Works SSL component pack (version 6) from /n Software. Although expensive for the one component we needed it for (TipsSOAPS), this contained everything we needed and we were able to complete the project.

This component pack is now up to version 9, and the only upgrade path is for the prior version (i.e. version 8). Given that we have Version 6 which doesn't work in Delphi XE3, it looked like we would need to purchase it again. But at $899, I wanted to see if there was another way!

Now don't get me wrong, it is a sophisticated and well supported component suite and I have no issue with the company or the product itself. But I am not prepared to pay full price for a product I already bought once, when that price is about the same as I paid for Delphi.

My first thought was to try to the Delphi Edition of IP*Works that is available to registered users of Delphi XE3. So I downloaded and installed it. I had a sneaking suspicion that this would not allow me to do what I needed, because there is a separate component pack for non-secure and secure communications. And indeed, the TipwSOAP component would not allow me to consume a web service over https:

Personally I think charging $899 to handle SSL URLs is a bit of a con.

So, knowing that Delphi introduced support for SOAP headers in Delphi 7, I knew that the only reason we had bought this product in the first place was no longer a problem. So I set about writing a prototype to see if I could implement the same functionality using native Delphi components.

And I have! So +1 for Delphi XE3. I just saved the company $899, and shed another third-party component pack at the same time (and that makes me very happy).

I should also point out that this is the first time I have really, properly used a Delphi IDE other than Delphi 6, and it was an huge pleasure. Very fast and productive. So far, so good!

Migration from Delphi 6 to Delphi XE3

Welcome back! It's been 2 and a half years since I last posted on this blog. Although it is partly down to the usual "too much work" and "kids keep me busy", it's largely because even after all this time, I'm still on Delphi 6(!), and so had nothing new to report. That's right, we never managed to find the time to migrate to Delphi 2009.

Not for much longer though. We've taken the decision that in March/April this year, myself and the development team are going to take the plunge, and finally upgrade from Delphi 6. Not to Delphi 2009 as I originally wrote about though. This time, we're going straight to the very latest version, Delphi XE3.

This has come about not only because our hand has been forced by some of the third-party components we use (TMS Component Pack, ReportBuilder, Raize Components) - none of the new versions of these support Delphi 6 - but also because we're starting to think about mobile development. Delphi XE3 offers decent distributed database architecture with DataSnap which we can make use of to write some terrific companion mobile apps for our software.

It is somewhat of a relief to see that dbGo (ADO) is still well supported with Delphi XE3, vindicating the decision I made a few years ago to switch over to that from ODBCExpress. This will form the largest part of our migration process, as we have to move all existing components over to the dbGo equivalent. However, we have established how to do this without any real complications, and so it will just be more time-consuming than difficult.

Of course, there will still be the Unicode effect that came when Delphi 2009 was released, "breaking" existing code. But this has been well documented by Delphi luminaries such as +Bob Swart, +Marco Cantù and +Cary Jensen, and so this will present no real issues.

The superb ReportBuilder Enterprise by Digital Metaphors stores the database connectivity information within the report file itself, so I have already written a utility that replaces the ODBCExpress details with that of ADO. This will instantly convert our 250 standard reports, as well as the countless reports written using the built-in report writer by users for themselves.

The largest task at hand is converting our "plug-ins". When I first joined the company over 14 years ago, my task was to rewrite the software to convert it from Unix to Windows. There were multiple versions of the software for each customer, based on their bespoke requirements. I vowed this would not be allowed to happen to the Windows version of the software, and so developed the plug-in architecture which allows bespoke functionality to appear within the core software, as if it were standard. To this day it works perfectly, however its success has grown much larger than I ever anticipated, meaning there are now over 100 plug-ins for various customers and other "extra" features that we can provide. Each of these is a separate Delphi project, and so this will be enormously time consuming to migrate these.

But it will be worth it. We will finally be on a development platform that can provide us with "modern" features, whilst giving us opportunities to develop in other areas as well.

I'll keep posting as I come across any interesting areas during the move! I'd love to hear from you if you have done anything similar and have any "gotchas" you'd like to share.