GSoC 2020 – Second Eval Status Update

We recently completed the second evaluation of our Google Summer of Code program. We’ve collected brief summaries of work from each of our 5 students, as well as the pull requests that our students have produced. Check them out below:

  • Giannis Gonidelis – Adapt HPX range-based parallel algorithms in order to expose a range based interface according to latest Ranges
    • During my second period at GSoC with Ste||ar Group I started working on the actual adaptations of the parallel algorithms. I adapted the for_each algorithm along with the transform algorithm. Along the adaptations there are certain traits that are being implemented along with other supporting utilities like the algorithms’ result types. What’s more important though, is that we decided with my mentors that we should broaden our target. We re-established our goal and decided that what HPX needs more is a general adaptation on the C++20 standard (and not just iterator-range based ones). Thus, the renewed goal is to provide for each algorithm 3 (2+1) versions: One conforming to the std namespace and one according to the std::ranges namespace (the 3rd version is the obsolete one and is marked as deprecated). 
  • Mahesh Kale: Pip Package for Phylanx
    • In the second phase of the GSoC project, a wheel package is generated that can be installed on any Linux Operating System that supports the cp37-cp37m tag. The wheel on testing with python tests in Phylanx passes 99/102 tests. 2 of the test cases fail due to misspelled import statements. The wheel is tested on fresh Ubuntu and fedora container. A CI is created that can produce cp37-cp37m and cp38-cp38 wheels.
    • Blog post
    • https://github.com/git-kale/phylanx_wheel
  • Pranav Gadikar: Domain decomposition, load balancing, and massively parallel solvers for the class of nonlocal models
    • Completed distributed implementation of the 2d nonlocal heat equation with a decomposition of the domain into smaller squares. Each square’s values for the numerical solution would be computed on a single node. However, each node can have multiple squares.
    • Completed the integration of GMSH and METIS libraries to obtain an efficient domain decomposition of the mesh. The main idea is to partition the coarse mesh i.e. the mesh created using bigger squares using the METIS library.
    • Working on load balancing of the distributed implementation 2d nonlocal heat equation. Currently, there is no load balancing algorithm readily available for the asynchronous many task systems. So the work involves devising a new load balancing algorithm and demonstrating it’s correctness on the use cases relevant to the 2d nonlocal equation.

GSoC 2020 – First Eval Status Update

In the first week of July, we completed the first evaluation of our Google Summer of Code program. We’ve collected brief summaries of work from each of our 5 students, as well as the pull requests that our students have produced. Check them out below:

  • Giannis Gonidelis – Adapt HPX range-based parallel algorithms in order to expose a range based interface according to latest Ranges
    • Pull requests
    • Working on adatpting the iterator-based algorithms in order to accept for a different type of begin() and end() iterator. This is an intermediate step in order to achieve the final goal to create as many range algorithms as possible. So the iterator-based ones shall work as a hidden layer which will expose the range functionality. Until past week, Giannis was working on these traits that needed to be implemented or adapted in order for them to be used on his first algorithm adaptation which is the for_each.hpp.
  • Weile Wei:  Concurrent Data structure Support
    • In the first month of our GSoC project on Concurrent Data Structure, we have initial HPX-threading support for hazard pointers in LibCDS library. On the HPX library side, we have created several hpx user-level thread-private data to support libcds, and on the LibCDS side, we have enabled threading backend from hpx threads. The following links demonstrate our work.
  • Mahesh Kale: Pip Package for Phylanx
    • In the first phase of our GSoC project, We have created a wheel package for Phylanx. The binary dependencies of the wheel, such as Boost, Blaze, HPX, etc., are glued inside the package following the bumpy directory structure. The wheel is self-contained and has no dependency on system libraries. It can be installed and imported on the Linux operating system and been tested thoroughly on Ubuntu.
    • https://github.com/git-kale/phylanx_wheel

GSoC 2020 Participants Announced!

It is time to announce the participants for in the STE||AR Group’s 2020 Google Summer of Code! We are very proud to announce the names of those 5 students this year who will be funded by Google to work on projects for our group.

These recipients represent only a handful of the many excellent proposals that we had to choose from. For those unfamiliar with the program, the Google Summer of Code brings together ambitious students from around the world with open source developers by giving each mentoring organization funds to hire a set number of participants. Students then write proposals, which they submit to a mentoring organization, in hopes of having their work funded.

Below are the students who will be working with the STE||AR Group this summer listed with their mentors and their proposal abstracts.


Participant:

Sayef Azad Sakin, University of Arizona

Mentor:

Kate Isaacs, University of Arizona

Project: Time Series Updates for Traveler

Traveler-Integrated is a platform for performance visualization of parallel runtimes such as HPX. The interface allows accessing multiple datasets, from different executions. However, there are several improvements that can be made to the interface in terms of managing large numbers of files (e.g., from historical regression runs) and organizing windows to handle comparison of runs. In this project you will work on the Javascript front end to implement this interface, refining the design through user feedback.


Participant:

Giannis Gonidelis, TS Aristotle University of Thessaloniki

Mentor:

Hartmut Kaiser, Louisiana State University

Thomas Heller, Exasol

Project:  Adapt HPX range-based parallel algorithms in order to expose a range based interface according to latest Ranges

Our parallel algorithms currently expect for the begin iterators to have the same type as the corresponding end iterators. The latest changes to the C++ Standard allow for those to be different. This enables the use of the algorithms with more general range abstractions (i.e. begin-iterator/sentinel pairs such as C-style strings). We have created a list of algorithms that need to be changed to support different types for the begin/end iterator pairs (see here). There you can also see what algorithms have already been adapted, which may help in understanding the task.


Participant:

Pranav Gadikar, Indian Institute of Technology Madras

Mentor:

Patrick Diehl, Louisiana State Univesrity

Prashant J Kumar, UT Austin

Project:  Domain decomposition, load balancing, and massively parallel solvers for the class of nonlocal models

Peridynamics is used to model cracks and fractures in materials. In recent years there have been several numerical approximations proposed for peridynamics. Peridynamics is a nonlocal model, i.e. to compute force/strain/energy at a point, one has to look at the neighbors of a point within a finite radius. A code utilizing the parallel algorithms and futurization within HPX for a single shared memory node is available. This code should be extended with a domain decomposition for computations on several nodes. Here, the challenge would be to perform an efficient load balancing of the domain partitions. In case of damage, the computational costs in the domain where the damage occurs in the material decreases. In this project, an efficient algorithm which detects where the computational costs are decreasing and how to redistribute the domains such that load is balanced should be developed.


Participant:

Weile Wei, Louisiana State University

Mentor:

John Biddiscombe, Swiss National Supercomputing Center

Mikael Simberg, Swiss National Supercomputing Center

Project:  Concurrent Data structure Support

STL containers such as vectors/maps/sets/etc are not thread safe. One cannot safely add or remove elements from one of these containers in one thread, whilst iterating or adding/removing in another thread without potentially catastrophic consequences (usually segmentation faults leading to eventual program failure). Some work has begun on implementing concurrent structures in HPX, a concurrent unordered map with reader/writer lock and a partial implementation of concurrent vector exist, but they have not all been completed, do not have unit tests and need to be unified into an hpx::concurrent namespace. A number of libraries implementing thread safe (sometimes lockfree) containers already exist that can be used for ideas and where code uses a boost compatible license can be integrated into HPX. The aim of the project is to collect as much information and as many implementations of threads safe containers and create or integrate them into the HPX library.


Participant:

Mahesh Kale, Indian Institute of Technology Roorke

Mentor:

Nikunj Gupta, Indian Institute of Technology Roorke

Parsa Amini, Louisiana State University

Project:  Pip Package for Phylanx

Phylanx relies on many external libraries which makes the building process tedious and error prone, especially to the target audience of the software- domain scientist. The goal of this project is to automate the build process of Phylanx by creating a distribution package. The distribution package should build and install Phylanx and its requirements through Python’s pip package manager.

HPX Community Survey

The HPX Community Survey was conducted to gauge user experiences regarding usefulness and functionality, purpose of use, and ease of use of HPX.  Submissions were collected through 03/05/2020, and 33 total responses were received.

The findings in this survey will serve as a valuable tool as HPX – it’s breadth and functionality – continues to grow.  HPX administrators and the newly instated PMC will consider the concerns presented in the results as system changes are made. The positive input is also very useful, pointing to processes that should remain consistent moving forward.

Basic Use

Poll results indicate the majority of respondents, 73%, currently use HPX. When asked where users first heard of HPX, though the results were many, nearly half of those polled were informed through a colleague or at a conference.  Other responses varied, but included The STE||AR website, YouTube, and Reddit. Though 21% of those polled knew of the existence of HPX for as long as 5-10 years ago, only 12% have been using the system for that long. With another 12% having never used HPX at all.

According to the survey, task-based programming is the main feature of HPX engaged by users, at 96%.  Distributed HPX and Parallel algorithms are also widely used at 64% and 54% respectively.  The least used feature, at 3.6% is Task to Hardware binding. The majority of respondents employ HPX for work-related projects and tasks at 83%.  Hobbyists tally in at 35% of those polled.Those employed by Academic institutions and students make up 50% of HPX users.  36% are in private industry with less than 10% self-employed.

Usefulness, Understanding and Changes

When asked to rank the usability of HPX on a scale of 1-5, with 1 being “Unusable” and 5 being “Great,” 82% of respondents rated a 4 or 5. And when asked on a scale of 1-5 whether HPX is getting better (5) or worse (1), 89% of those polled selected 4 or 5. 

Respondents were asked what they most appreciate about HPX; many remarked on HPX’s C++ standardization and compliance. One commented, “Great technology with good API. Standards compliance. Parallelization made easy.” Conversely, when asked what they liked least, there were a few who also stated C++.  One comment simply being “C++ is a horrible language.”  As it were, there were other concerns, several regarding lack of documentation, others on complexity or difficulty integrating with an existing application. 

These concerns went hand-in-hand with the many of the comments pointing to improvements that could be made.  Requests for more complete documentation being a repeated concern.  Other suggestions included offering additional and optional safety features or a more intelligent scheduler. When asked why a user might stop using HPX, one again pointed to lack of documentation while another pointed to the difficulty in learning how to work within the system.

Those polled were next asked to rate HPX’s documentation across several topics on a 5-point scale, with 5 being excellent.  The topics were as follows:

  • What does HPX do?
  • Getting started, compiling, installing
  • Task-based programming
  • Parallel algorithms
  • Components, actions
  • Debugging
  • Profiling
  • Low-level functionality
  • API reference
  • Examples and tutorials

Of these, topics the first 4 scored best with an overall “Good”/4 rating, with “What does HPX Do?” and “Getting started, compiling, installing” having the highest number of “Good” responses.  The first 4 topics also received the largest number of “Excellent”/1 responses, though not a majority. Conversely, topics “Debugging,” “Profiling” and  “Low-level functionality” received a majority of “Severely Lacking”/2 ratings.  These 3 topics were also given a few 1 ratings, meaning users could find no documentation on these topics at all.

Regarding specific items that users would like to see documented, responses varied greatly.  Two users polled requested documentation and support for the parcelport system.  Another stated that documentation on integration with existing code would be beneficial.  One user suggested that the need for documentation varies depending on the task and the week, typically on low-level topics.

HPX Community

When asked if users have felt welcomed into the HPX community, the response was overwhelmingly “Yes,” at 80%, with no “No” responses.  The remaining 20% haven’t formed an opinion on this question yet. Results were split across categories when asked a preference of where to have the HPX community.  Answers included: A mailing list, IRC, Slack, Matrix and Discord. 

Regarding how users interact with each other within the HPX community, the majority, at 68%, is through GitHub, with other popular responses being through the IRC channel on Freenode or through direct contact with developers. In response to whether the IRC channel is easy to join, most replied “Yes,” at 40%, another 33% did not care, and only 7% experienced difficulty and did not join because of it.

Contributions to HPX

Of those polled, 41% of users have contributed to HPX on multiple occasions.  28% have contributed once.  Of the remaining 31%, some would like to contribute but cannot find time, others find HPX too intimidating to contribute, and still others have no interest.

When asked what changes specifically would make contributing easier for users, some again pointed to additional documentation.  Another request was for “Smaller tasks for first timers.”  One other user pointed back to the previous response of too little time personally, no matter what changes might occur in the system.

HPX users were asked how they might contribute to HPX in “non-code” ways.  Responses were few but included “documentation,” and starting an “HPX study group or meetup for intermediate C++ programmers.”

The final question asked respondents to share any additional information they choose.  Here is one user’s response: “All in all a very good piece of tech. Needs more approachability, especially work on documentation in many ways. Keep up the work. Despite all shortcomings I love it!”

We’d like to thank all those who took the time to participate in the survey; please know that your opinions and responses are important to us.  HPX could not function without the community built around it.  If you have any questions about the survey or further concerns, please contact us at hpx-users@stellar.cct.lsu.edu

GSoD Final Report

By Rebecca Stobaugh.

We’ve reached the end of Google’s Season of Docs, and we’ve accomplished a lot in the past three months. My initial proposal was to work on three sections of the manual, and we have far exceeded our goal, managing to make changes to twelve different sections of the documentation. The majority of the work I’ve done has consisted of cleaning up grammatical errors and improving sentence structure. I have also added a style guide to the wiki, which should help standardize future changes to the documentation. The style guide can be found in the “HPX Source Code Structure and Coding Standards” wiki document under the section “Documentation Style Guide”. For a complete list of my pull requests during Season of Docs, please see here. To view my changes to the wiki, please see here.

Announcing HPX’s Season of Docs

By Rebecca Stobaugh

HPX was recently selected to be part of Google’s Season of Docs (GSoD), a program designed to improve the documentation of open source software. While the STE||AR Group has created extensive documentation for HPX, this documentation has been written by several different people, which has led to some inconsistencies and awkward organization. I am a technical writer and English PhD student who has been selected to edit and streamline HPX’s documentation. My goal is to clean up the content, concentrating on both grammatical issues and design concerns, to create a more cohesive, user-friendly product. My primary focus will be on two sections of the STE||AR Group’s instruction manual: “HPX Build System” and “Launching and Configuring HPX Applications.” If time allows, I will also revise the “Why HPX” page, with an emphasis on condensing and trimming repetitive content.

You can read my GSoD proposal here.

Trip Report: ICML 2019


By: Bita Hasheminezhad

A few weeks ago, I had the opportunity to attend the International Conference on Machine Learning (ICML) which is the premier gathering of professionals dedicated to the advancement of machine learning. Thirty-Sixth ICML was held on June 9th to 15th at the Long Beach Convention Center.

HPX 1.3.0 Released!

The STE||AR Group is proud to announce the release of HPX 1.3.0! This release focuses on performance and stability improvements. Make sure to read the full release notes to see all new and breaking changes. Thank you once again to everyone in the STE||AR Group and all the volunteers who have provided fixes, opened issues, and improved documentation.

Download the release from our download page, or GitHub page.

HPX accepted for Google Season of Docs 2019

This year Google is organizing for the first time Google Season of Docs (GSoD). Like Google Summer of Code (GSoC) the program aims to match motivated people with interesting open source projects that are looking for volunteer contributions. GSoD, however, aims to improve open source project documentation, which often tends to get less attention than the code itself. We recognize this all too well in the HPX project. For this reason, we decided to apply for GSoD and can now proudly announce that HPX has been selected as one of 50 projects participating in this year’s GSoD!

This means that we are now looking for motivated people to help us improve our documentation. If you have some prior experience with technical writing, and are interested in working together with us on making the documentation of a cutting edge open source C++ library the best possible guide for new and experienced users, this is your chance. You can read more about the program on the official GSoD home page. We’ve provided a few project ideas on our wiki, but you can also come up with your own. Our current documentation can be found here.

The deadline for technical writer applications is June 28. Come talk to us about your ideas and your application on our mailing list, IRC, or Slack. We’d love to hear from you!