Import/export nightmare

First of all I apologize for the lack of writing in the last weeks. I wish I could find the time to write more regularly. I'm still working hard on STLport for Symbian OS, and I'm also building my own website where I plan to host development projects (included STLport), some of my work on OpenSceneGraph, a section dedicated to technical articles, and more. This blog will also move to the new home when it's ready.

One of the main requirements for the second release of STLport for Symbian OS is to provide a DLL build target in addition to the already provided static library target. The common reasons for building a library as a DLL are well known, but the main reason for doing this on Symbian comes from an unwanted "feature" of the build chain provided with Carbide.c++ Express: it seems that if you have a static library a.lib and another static library b.lib that depends on the former, trying to build an application that uses b.lib results in linker errors because symbols defined in a.lib that are used in b.lib are considered as undefined, even though they should be found because both libraries are being linked together. In other words, it seems that Carbide.c++ checks static libraries for undefined symbols once at a time without considering the "big picture" of all libraries being linked together. This of course makes the build fail because dependencies between static libraries always result in linker errors, hence the need for a DLL build target.

So, what's the difficulty in providing a DLL build target? Unfortunately, it's not just matter of modifying build options. One has to mark every function and variable declaration with the IMPORT_C macro, and their definition with the EXPORT_C macro. Usually, large projects like STLport already contain some sort of dllexport/dllimport macro system, but the way it works in Symbian is (once again) radically different from other systems, so you can't rely on it. In a nutshell, here is the difference:

Windows - MSVC

• The implementation (definition) is never flagged with import/export macros
• The interface (declaration) is flagged with a macro that assumes an "export" meaning when the library is being built, and an "import" meaning when the library is being used by client code
• When a class is flagged, all of its members are automatically imported/exported

Symbian - GCCE/WINSCW

• The implementation is always flagged with EXPORT_C
• The interface is always flagged with IMPORT_C
• All class members that need to be imported/exported must be flagged individually

So, even though the goals are similar if not the same, the approaches are different and not compatible. This means that a lot of manual work is potentially necessary to make STLport a Symbian DLL, especially if you are not very familiar with the code (and I'm not).

Given my commitment to port other libraries (like Boost) to Symbian, I've decided to invest some time to find a way to do the "dirty work" automatically by analyzing source files and identifying those declarations and definitions that need to be imported or exported. This is a tough task that is driving me nuts. I initially tried with Doxygen's XML representation, but I found it inconsistent when it comes to parse heavily templatized code like that of STLport. Then I tried GCC-XML but it seems to be using an old version of GCC that can't parse STLport's source files correctly.

I'm currently experimenting with a C++ front-end from EDG which is provided in the Program Database Toolkit package from the University of Oregon. Unfortunately, I can't use the intermediate language representation produced by the EDG front-end because it's in binary format and its documentation is not included in the PDT project (the front-end is a commercial product, and its internal documentation can't be redistributed). The front-end also generates a cross-reference file which is documented, and that's the file I'm currently experimenting with. It doesn't contain all the information I need on declarations and definitions, but I give it a try.

Development of STLport for Symbian is all but dead. I just hope to solve the above problems soon. I wish to thank all of you who have sent me feedbacks in the last weeks, I really appreciate it. Please keep testing STLport!

What's next?

What's coming after the release of STLport for Symbian OS? Well, at least a second release will follow shortly, possibly including a full implementation of STL locales and a fix for the missing typeinfo header in the platform SDKs. I'm also considering the possibility of building STLport as a DLL, or as a set of DLLs if a single binary is too large. This will help reduce the size of application executables, saving precious space on the device's memory.

The STL is just my first step towards a better development platform on Symbian OS. The second step will be longer and probably more difficult: porting Boost libraries. If you don't know what the Boost project is, please have a look here: http://www.boost.org. I think every serious C++ programmer should know Boost, regardless of whether he/she decides to use it or not. It's a matter of fact, some developers love Boost and others just can't live with it. I personally love it, in my opinion it is the perfect complement to the STL and to the C++ language itself. The reason why some programmers don't like it is that it is radically different from other libraries, which are usually based on (and often abuse) "ordinary" OOP paradigms and idioms. In contrast, Boost libraries try to enforce those features that are unique to the C++ language, such as compile-time polymorphism and template metaprogramming, resulting in higher-performance elegant code that can be extremely reusable if designed well. The downside is that using Boost is usually more difficult than using other libraries, and that's especially true for developers who don't write C++ programs "the C++ way". If you don't know what I mean, I suggest you to read this excellent book by Herb Sutter and Andrei Alexandrescu: C++ Coding Standards, a must for every C++ programmer.

Of course, neither the STL nor Boost libraries can replace the Symbian API and its nonstandard set of paradigms, rules and guidelines, but they can serve as a base for building an interface layer to abstract away the intricacies of Symbian programming and finally make writing software for this platform easier, safer and faster.

These are my long-term plans. Maybe they're too ambitious, given my lack of spare time...

STLport for Symbian OS released!

STLport for Symbian OS has been updated! A pre-release of version 5.1.5 (source code only) is available here.

I'm happy to announce the first full-featured release of STLport for Symbian OS 9.x!

STLport is a high-quality implementation of the C++ Standard Template Library (STL), a library that is part of the ISO/IEC C++ Standard and that is not included natively in the Symbian OS platform. By supporting the STL, Symbian applications can benefit from a wide range of existing C++ code that can now be ported easily.

This version of STLport should work on all Symbian OS 9.x devices, both S60 and UIQ variants. The library has been succesfully compiled with (and binaries are provided for) following platform SDKs:

• S60 3rd Edition 1.1 (Maintenance Release)
• S60 3rd Edition Feature Pack 1
• UIQ 3.0

The only dependency is a conforming Standard C library, that is provided by either the P.I.P.S. library (available for both S60 and UIQ devices) or Nokia's Open C plugin (S60 only).

All features of STLport have been ported, although locales other than the default "C" locale are not yet implemented. This will be done in next releases. All unit tests have passed on both the emulator and a wide range of devices, while only one test in the exception handling testsuite is still failing. More details are available in documentation files that come with the downloadable package.

So, now the interesting part. If you are using one of the aforementioned SDKs and you don't need to change any of the STLport configuration options, then you can download the prebuilt package that includes header files and binaries for a specific SDK:

Download prebuilt package for S60 3rd Edition 1.1 Maintenance Release SDK

Download prebuilt package for S60 3rd Edition Feature Pack 1 SDK

Download prebuilt package for UIQ 3.0 SDK

Download the README file that contains installation instructions

Please note that binaries for S60 were compiled against the OpenC library, while those for UIQ were compiled against the P.I.P.S. library. If you want to compile the library for another SDK or if you want to change the build configuration, you need the full source code so that you can recompile STLport from scratch:

Download full source code

The source code package also contains the necessary files for building an installable SIS package containing the STLport test suite.

If you have any questions or suggestions, please let me know. I can be contacted on the Forum Nokia's discussion forum, my nickname is "polo78".

Enjoy!

STLport coming tomorrow for both S60 and UIQ!

The first release of STLport for Symbian OS is ready, and the big news is... It works on UIQ 3.0 too! The only practical difference is the standard C library, which is Symbian PIPS on UIQ and Nokia's OpenC plugin on S60. These two libraries are actually very similar and have some parts in common, but whereas OpenC includes a "stdio server" for displaying and/or redirecting standard streams, PIPS apparently doesn't have such feature. This means that C-style console applications based on PIPS can't display messages through functions like printf(). Initially, this limitation was an issue because I couldn't run the STLport unit test program to test the library on the UIQ emulator or on the device (I've bought an used M600i phone from eBay), but I solved it by writing a simple shell application that launches the STLport test programs and captures their output.

So, everything is in place. There is a good chance that I'll be able to make the release available for download by tomorrow night.

STLport and the problem of destructors not being called in Symbian OS

While I was away I did some additional testing on my STLport variant for Symbian OS and I noticed an annoying fact: everytime a GUI application terminates, even gracefully, an ALLOC panic is raised on the emulator indicating a possible memory leak (which is a rather common bug in programs). It turned out that STLport uses a sort of optimized allocator to manage memory allocation and deallocation for STL containers and other classes. Such allocator is a global object that pre-allocates memory on its own strategy, but it doesn't free all allocated blocks when the program ends (I think that's because the operating system is supposed to do the work). This is not a problem under normal conditions, but Symbian GUI apps usually try to detect memory leaks by marking the heap and tracking memory usage throughout the program execution, thus raising an ALLOC panic when the STL-based applications ends. Although this is benign and doesn't happen on the real device (memory leaks are only detected in debug builds), it makes the useful memory leak check completely useless, which is bad.

After a bit of investigation I discovered a configuration option in STLport that forces the optimized allocator to release all allocated memory on program termination. I tried the same test with that option turned on, but nothing changed. Apparently, the reason is much more serious than I initially thought: it seems that class destructor of global objects in Symbian applications is never called, even though it is supposed to be called (as per the C++ ISO/IEC Standard) when the program exits cleanly. In the case of STLport, this means that memory still owned by the optimized allocator is not released on exit. This is a serious malfunction of the Symbian application model in my opinion.

I asked other developers at Forum Nokia an explanation for this fact, with no luck. Some users confirmed that destructors are not being called, but neither a motivation nor a workaround emerged from the discussion. For now, I'm simply switching off the optimized allocator in STLport, falling back to the "direct" malloc/free allocator, which may be a bit less efficient but at least it doesn't leave allocated memory around.

Back to work now, more news soon.

Native Symbian fstream implementation for STLport is done

As I mentioned in my previous post, the first attempt of porting the fstream component of STLport to Symbian OS using plain stdio functions (provided by OpenC) didn't produce good results. Performance was far from being acceptable, and the library had serious stability issues.
I've now implemented the fstream core in terms of native Symbian OS calls (RFs, RFile and the like) and the overall performance has sensibly improved. I don't get panics or strange behaviours anymore. Of course I can't guarantee it's 100% safe, as dealing with STLport code is not an easy task, and I'm a novice Symbian developer. Anyway, my implementation has passed all STLport regression tests, so I'm pretty optimistic.

The good news is, now all 385 unit tests pass on both the emulator and the device!

Today I've spent a lot of time trying to compile a simple "hello world" program based on STLport from the Carbide.c++ 1.1 Express IDE, with no luck. Although the app compiled and linked just fine from the command line (ABLD), the Carbide build process resulted in compiler failures (presumably caused by incorrect order of include paths) and mysterious linker errors. Fortunately the new Carbide.c++ 1.2 is now available for download (if only I had discovered it yesterday!). I'm pleased to say that this new version is much more robust and reliable than the previous one. More important, my STL-based "hello world" application builds and runs perfectly with Carbide.c++ 1.2.

So, I'm now in the final stages: recompile everything, double-check everything, write some README.blah.blah files and make the release. This is going to happen next week, however, as I'm taking a short holiday on Labour day.

Update on STLport for Symbian OS

This is an update for those of you who are following the development of STLport for Symbian OS in this blog. As I announced in my previous post the library is almost complete, but I've found that I/O through fstream is pretty unstable, especially in the emulator. After fixing a small bug in STLport I got the stdio-based implementation of fstream work, although with poor performance, on my N80 phone and on a few other devices through Remote Device Access. However, performance on the emulator is so horrible that I couldn't make fstream unit tests complete in a reasonable amount of time. Also, the emulator occasionally crashes or terminates the application with a panic when certain combinations of read/write operations are done.

I think the I/O routines in OpenC are quite buggy (as seems to be the family of printf functions, see here) because even simple code can occasionally fail or even crash. In particular, I'm observing a behavior similar to that described in this post at Forum Nokia, but it's not easily reproducible as it doesn't happen always (and not always under the same conditions).
As a result of two days wasted trying to make buggy things work, I've decided to try a better way: provide a custom implementation of fstream based on native Symbian OS calls rather than on OpenC calls. This shouldn't be too hard, but if I see that it's going to take too long then I'll just keep things as they are and I'll defer the fix until the next release.

Current status of unit tests is pretty encouraging though: only 3 tests out of 385 are still failing! I'm confident that I'll be able to make a release before next week.

Porting the C++ STL to Symbian OS 9.1

Over the last three days I've been working on adapting STLport, an open-source implementation of the C++ Standard Template Library (http://stlport.sourceforge.net), to the Symbian OS. My goal is to port all features of the library, including those parts that were deliberately ignored in previous porting attempts due to limitations of older Symbian systems, such as iostream, fstream and locale.

The standard C library that comes with the new OpenC plugin for S60 is the perfect ground for building STLport, which doesn't require any dependencies other than a decent compiler (both WINSCW and GCCE work perfectly) and a standard-compliant C library. Please note that I'm developing with S60 SDKs only, I don't know whether other kits (such as UIQ) come with a suitable C library or whether OpenC would work on such SDKs. So, for now, the port will be limited to S60 3rd edition.

I'm happy to say that the porting process is close to completion. I managed to build almost everything of STLport on both WINSCW and GCCE platforms and I'm now running unit tests and fixing things as I get errors. I think I've even found a bug in STLport that prevents fstream from working correctly (details here: http://sourceforge.net/tracker/index.php?func=detail&aid=1704843&group_id=146814&atid=766244). The only big part still missing is the support for locales, as it requires a customized implementation. Once I get the library up & running I'll dedicate some time to implement it, in the meanwhile a "dummy" locale implementation is present in order to avoid breaking existing code.

So, what are the motivations behind my decision to port the STL to Symbian? Well, the most obvious one is portability. I've written a lot of C++ applications in the past, especially non-GUI engines and OpenGL-based 3D apps, and all of them are based on the STL. Porting them to a new system/platform can be either a breeze, if the new development environment supports the STL, or a pain, if it doesn't. Moreover, sharing a single code base between different platforms can cut maintenance costs down to about half, which is good.

The second reason is that programming for Symbian is difficult. Antony Pranata once wrote an interesting post on this topic, see http://mobile.antonypranata.com/2005/12/19/why-is-symbian-os-development-very-difficult/. The STL, on the contrary, uses widely-recognized programming paradigms and, even better, it enforces good paradigms by exploiting the C++ language itself. Symbian libraries don't enforce the programming style they require, and I think this is the main reason (beyond the horrible, IMHO, coding style) that keeps novice Symbian developers away from the field. Such programmers are given too much responsibility (i.e. manage the cleanup stack manually) and they need to take care of too many aspects that don't have anything to do with the useful code they want to write. When they try and fail too many times, there is a good chance they will give up and never get back to Symbian again unless they have to.

I'm not going to start a flame about the quality of Symbian code. I'm pretty sure there are many experienced developers out there that don't find it difficult to write Symbian programs, but I also think that it's perhaps even more important to "open the doors" to new developers by making things easier and more standard-compliant. I do recognize the reasons behind Symbian architects' choices: when the OS was designed, the C++ language wasn't even standardized and lacked several features like exception handling, so they had to invent their own mechanisms (leave/trap, cleanup and so on). But come on guys, that was 20 years ago...

Anyway, back on topic. I think I'll be able to publish the final port of STLport for S60 3rd edition in few days. I'm going to release both source code and binaries for WINSCW (the emulator) and GCCE (the real phone). Using STLport will be of course possible from any development environment, including command-line (ABLD) and the Carbide.c++ family of products. Initially I'll compile STLport in the form of a static library, because the resulting binary is very large (8 MB) and making it a DLL would be overkill.

More news soon!

Shortcut icons for hidden applications in S60 devices

Some Nokia smartphones based on the Symbian/S60 platform come with a set of preinstalled applications dedicated to the acquisition and management of GPS data. In most of these devices, however, such applications are "hidden", that is they don't have an associated menu item. This is especially true for N-series devices like the N80, while Nokia's business-line phones (E-series) do include shortcut icons in their menu.

This is a well-known fact that has been discussed many times in the past months. The usual procedure for running one of the hidden applications involves using a task manager application to first find the right executable within the list of installed applications and then execute it. This is quite tedious if you need to do that often, so I've put together a tiny installable package that adds the missing shortcut icons to your phone's Tools menu, as you can see in the following picture taken from my N80 (sorry it is in Italian):

The icons added by this package are, in order:

• Landmarks
  A database of geographic locations that is shared among applications;
• Positioning methods
  System-wide configuration of positioning methods that are provided to applications;
• Navigator
  A tool that displays GPS data such as latitude, longitude, altitude and speed; it also provides basic navigation features that may be useful for maritime or sports use.

Please note that not all phones have these three applications installed. For example, the N73 only has Landmarks, while the N93 doesn't have any of them. Only icons for those applications that can be found on your device will be added to the menu. I've succesfully tested the package on these phones:

• Nokia 3250 (Landmarks only)
• Nokia 5500 Sport
• N73 (Landmarks only)
• N80

Icon labels are localized in two languages: Italian and English.

You can download the file here, it's a self-signed SIS package: http://www.webalice.it/marco.jez/files/navlauncher.sis

Enjoy!

Welcome!

Dear reader,

welcome to my blog! If you're interested in computer programming then you've come to the right place. In these pages you will find news, tips & tricks, code snippets and other features about the exciting world of software development. Most of this blog's posts will focus on the C++ language and related topics, but that's not a strict requirement. If you're not a developer but you are a technology enthusiast you may still want to land here from time to time as I occasionally write for a broader audience too. In this blog you will also find announcements about software applications that I write in my spare time and that I decide to release to the public, so stay tuned!

Now a few words about myself. My name is Marco, I live in Italy where I work as a C++ programmer for a small company that produces custom software and content for real-time 3D visualization systems. For the most part my job is about writing interactive 3D applications using OpenGL (http://www.opengl.org) and higher-level libraries like OpenSceneGraph (http://www.openscenegraph.org), but as a hobby I like to explore other fields too, such as web applications and mobile devices.

Why a blog, you may ask? Well, in the past years I've been actively following several mailing lists and discussion forums where I used to share my everyday work experience with other mates. Unfortunately this is a time-intensive activity that requires a relevant amount of free time. As my work commitment grew over the years I found myself having less and less spare time, so I've now decided to concentrate all my collaborative efforts into a single place. Such place is this blog, my own "experience container".

I hope you will find these pages useful for you. If you have comments, questions or suggestions, please let me know. Feedback is vital!