Wednesday 11 July 2007

Import/export nightmare

First of all I apologize for the lack of writing in the last weeks. I wish I could find the time to write more regularly. I'm still working hard on STLport for Symbian OS, and I'm also building my own website where I plan to host development projects (included STLport), some of my work on OpenSceneGraph, a section dedicated to technical articles, and more. This blog will also move to the new home when it's ready.

One of the main requirements for the second release of STLport for Symbian OS is to provide a DLL build target in addition to the already provided static library target. The common reasons for building a library as a DLL are well known, but the main reason for doing this on Symbian comes from an unwanted "feature" of the build chain provided with Carbide.c++ Express: it seems that if you have a static library a.lib and another static library b.lib that depends on the former, trying to build an application that uses b.lib results in linker errors because symbols defined in a.lib that are used in b.lib are considered as undefined, even though they should be found because both libraries are being linked together. In other words, it seems that Carbide.c++ checks static libraries for undefined symbols once at a time without considering the "big picture" of all libraries being linked together. This of course makes the build fail because dependencies between static libraries always result in linker errors, hence the need for a DLL build target.

So, what's the difficulty in providing a DLL build target? Unfortunately, it's not just matter of modifying build options. One has to mark every function and variable declaration with the IMPORT_C macro, and their definition with the EXPORT_C macro. Usually, large projects like STLport already contain some sort of dllexport/dllimport macro system, but the way it works in Symbian is (once again) radically different from other systems, so you can't rely on it. In a nutshell, here is the difference:

Windows - MSVC
  • The implementation (definition) is never flagged with import/export macros
  • The interface (declaration) is flagged with a macro that assumes an "export" meaning when the library is being built, and an "import" meaning when the library is being used by client code
  • When a class is flagged, all of its members are automatically imported/exported
Symbian - GCCE/WINSCW
  • The implementation is always flagged with EXPORT_C
  • The interface is always flagged with IMPORT_C
  • All class members that need to be imported/exported must be flagged individually

So, even though the goals are similar if not the same, the approaches are different and not compatible. This means that a lot of manual work is potentially necessary to make STLport a Symbian DLL, especially if you are not very familiar with the code (and I'm not).

Given my commitment to port other libraries (like Boost) to Symbian, I've decided to invest some time to find a way to do the "dirty work" automatically by analyzing source files and identifying those declarations and definitions that need to be imported or exported. This is a tough task that is driving me nuts. I initially tried with Doxygen's XML representation, but I found it inconsistent when it comes to parse heavily templatized code like that of STLport. Then I tried GCC-XML but it seems to be using an old version of GCC that can't parse STLport's source files correctly.

I'm currently experimenting with a C++ front-end from EDG which is provided in the Program Database Toolkit package from the University of Oregon. Unfortunately, I can't use the intermediate language representation produced by the EDG front-end because it's in binary format and its documentation is not included in the PDT project (the front-end is a commercial product, and its internal documentation can't be redistributed). The front-end also generates a cross-reference file which is documented, and that's the file I'm currently experimenting with. It doesn't contain all the information I need on declarations and definitions, but I give it a try.

Development of STLport for Symbian is all but dead. I just hope to solve the above problems soon. I wish to thank all of you who have sent me feedbacks in the last weeks, I really appreciate it. Please keep testing STLport!