GSoC 18: Final Documentation for a C++ run time replacement

By Nikunj Gupta

A C++ Runtime Replacement


A user can make use of the HPX functionality only after initializing the HPX Runtime System. To initialize the user has to call thehpx::init function with valid parameters and an entry point for the HPX runtime system (defaulting tohpx_main if none provided).

My GSoC project was to integrate the HPX runtime system with C++ runtime system i.e. to make all HPX features directly available from int main. To give a clear picture, this was already pre-implemented using a C++ macro #define main hpx_startup::user_main. This method, however, is not robust and can result in unexpected errors. The project, therefore, involved researching various ways to integrate the two seamlessly and make sure it is robust at the same time. I was able to integrate the runtime systems for machines running Linux or Mac OSX.

Research Work


I had a few constraints to follow to make the implementation robust. These are as follows:

  • I can not create a specific HPX toolchain to link to instead of standard libraries. While this method looks robust at first, it involves writing a lot of platform-specific code. Another problem will be maintaining the startup code by updating it frequently.
  • I can not use class objects to initialize the runtime system before main (using global objects). These are usually implemented
    using static function initializations which can result in a well known problem of static initialization order fiasco. While these can be removed with careful examination, the implementation as a whole cannot be termed robust.

With that said, I researched ways to implement the same and found various ways to implement the same with multiple pros and cons. These are discussed in brief in this blog. For further details, you can check out my previous blogs.

Phase 1

With the commencement of Phase 1, I began my research to look out for robust ways to integrate the two runtime systems. I
first searched out for ways to implement the same for Linux. At the same time I created a repository for experimenting with what I had learnt. The link to same can be found here.

I started off with searching ways to hook (changing the regular behaviour of the function according to our needs) into one of the functions present in the C calling convention (the startup functions of glibc which initializes the C runtime system and provides the entry
point as int main). With runtime hooking in mind, I stumbled upon the dlfcn library. It provided with functions that can help hook into a function at runtime (providing a dynamic symbolic link). This method worked seamlessly and passed all tests.

This method, however, had two major flaw (one of which was obvious, while the other was discovered recently). These are:

  • This method relied on dlfcn.h. This meant this method could only be used for dynamic executable and it will result in a segmentation fault for static executable.
  • This method relied on one of the definition of __libc_start_main which meant non portable implementation. The code would therefore break for platforms like PowerPC and XeonPhi.

In the meantime, I also built HPX on Windows to try and implement the same on Windows. I got to know more about the C calling conventions for MSVC compiled executable on Windows. I could not find any evidence of similar implementation on Windows. I was therefore assured by my mentor that he’ll put me in contact with someone from Microsoft to help me out with problems relating to Windows. Thus, I reverted to Linux implementation for a better implementation.

For function codes and examples, you should visit my previous blogs
here and

Pull Requests

  1. Replacing hpx_main’s #define main with overriding __libc_start_main (#3338) (CLOSED)
    • Implementation for dynamic executable on Linux
    • Closed due to drawbacks explained above

Phase 2

With major drawbacks in my Linux implementation I was forced to find a better solution. This was the time I started looking into the Linux linker (ld) for potential implementations. I digged deep and also contacted the glibc team at IRC. With self research and their help, I was able to figure out a way to implement the same for a static executable. The Linux linker provides with an option of –wrap. This can be used to wrap any function. To put it simply, a function wrapper is one at which all calls to the actual function are redirected to. I chose the function __libc_start_main to wrap (later to realize that it was a very bad decision). To know more about what this function does, check this link.

Once I wrapped the function, I changed the program’s entry point and initialized the HPX runtime system, later to call the main function manually. This function worked seamlessly and was robust as well (since it worked for both static and dynamic executable and didn’t contain any pitfalls for the macro implementation). One major drawback which I realized later (after the PR was merged!) was the platform specific definition of _libc_start_main. This meant that the code now behaved differently for different platforms. Therefore the easiest method to fix the issue
was to wrap the main function itself. The code worked perfectly once the changes were made.

The code was built as a static library which was made to link to every executable. Thus, I needed to make sure that this
implementation was only used when the header file hpx_main.hpp was included. This was implemented using weak and strong variables. The strong variable in hpx_main.hpp overrided the one present in the static library to determine the C calling stack at runtime.

At the same, I also researched about Mac OSX implementation. Since, I found a lead in Linux linker for Linux implementation, I went ahead and searched for something similar to –wrap. While the option, was not present I stumbled upon the -e option which changed the program’s entry point. Thus a similar implementation was made possible for Mac OSX implementation as well.

Upon talking to the Windows team, I was proposed with one way of implementation. The method was robust enough but the only
problem was limited functionality of HPX functions from main. I therefore tried to implement the mainCRTStartup (function similar to __libc_start_main), another NOT recommended option proposed by the MSVC team.

For codes and examples, visit my blog here and here.

Pull Requests

  1. Static/dynamic executable implementation (#3357) (MERGED)
    • Implements for Linux systems (only on x64, x86_64 platforms)
    • Introduces another static library (libhpx_wrap) that is built while building HPX
    • Function stack determined at runtime using shadow variables
    • Works for any Linux system using glibc to link to
  2. Adds Mac OS implementation to hpx_main.hp (#3367) (MERGED)
    • Extends the implementation to Mac OSX

Phase 3

The PR (#3357) on merging broke the master branch. This was due it’s platform specific implementation. I therefore made the changes accordingly to change the wrapping function. In the meantime, I also updated the HPX docs to include the recent implementation changes.

The PR (#3357) also lacked HPX application support (like Phylanx). Thus, Phylanx stopped building as well. To rectify, I added cmake export variables
HPX_LINKER_FLAGS and HPX_LINK_LIBRARIES. This way any HPX application will have to add those to build successfully.

Lastly, I added a debugging option to signal the user if he includeshpx_main.hpp and then try to initialize the HPX runtime system. This way, the user will be notified that the HPX runtime system was already initialized from main and that he needs to remove the header if he wishes to initialize it himself.

Pull Requests: HPX

  1. Updates doc with recent hpx_wrap implementation (#3368) (MERGED)
    • Updates docs to include the recent changes in HPX
    • Adds another section to HPX docs dedicated to the implementation
  2. Adds HPX_LINKER_FLAGS to HPX applications without editing their source codes (#3371) (MERGED)
    • Adds export variable HPX_LINKER_FLAGS (containing the required flags to link to
      libhpx_wrap) to CMakelists.txt
  3. Adds hpx_wrap to HPX_LINK_LIBRARIES which links only when specified (#3374) (MERGED)
    • Adds export variable HPX_LINK_LIBRARIES (containing
      libhpx_wrap) to CMakelists.txt
  4. Replacing wrapper for __libc_start_main with main (#3375) (MERGED)
    • Replaces wrapper function from __libc_start_main to main
    • Makes code portable to run on every platform
  5. Adds Debug option for hpx initializing from main (#3385) (MERGED)
    • Adds debug option for the implementation signalling when user tries to re-initialize the system after pre-initiation
      from main.
  6. Allow debug option to be enabled only for Linux systems with dynamic main on (#3402) (MERGED)
    • Fixes broken master my including header guards over the hpx_start::include_libhpx_wrap variable.
    • Updated later to include Mac OSX implementation as well.

Pull Requests: Phylanx

  1. Change Phylanx to build according to the pull request #3374 in HPX (#519) (MERGED)
    • Changes CMakelists.txt to build Phylanx according to the current master of HPX

Beyond GSoC

While contacting the Microsoft Team, I was informed that a similar implementation could not implemented on the Windows platform
(proposing other ideas). Therefore, I and my mentor concluded that I should skip the Windows implementation for now.
I, however, kept looking for solutions. I will carry on with this research. Along with this, I will change the core HPX
to allow for multiplehpx_init calls. This will be useful for applications with backends written in HPX.

I’ll also work on the init function specific to HPX. This function will be responsible to run any global object on an HPX thread (to completely integrate the HPX runtime with the C runtime system).

Finally, I will work on the documentation porting to Sphinx.