Modules

This page discusses how Quinoa's code repositories are organized into git submodules and subtrees. It also gives an example of how to add a new third-party library (TPL) as a new submodule.

Quinoa uses git submodules as well as git subtrees to pull in third-party libraries (TPLs) and auxiliary tools. This allows easy update to a TPL's (or tool's) new or custom version, record all TPLs' and tools' histories (including branches and tags), and also allows contributing fixes upstream, all within git. There are some differences between git's submodule and subtree functionality each with its own pros and cons. This page explains how to use git submodules and subtrees in Quinoa.

TPLs and external tools

An important distinction between TPLs and tools is that TPLs are libraries whose source we directly use, e.g., via include files and/or (static and dynamic) linking, while from a tool we do not include source code nor link its object files, only build it and use it as a separate (external) executable. To learn more about TPLs and tools, check out the page on Libraries.

Git subtree for sharing cmake code

Currently there is a single git subtree in quinoa and is used to reuse/share cmake code between two repositories

Both of the above git repositories are setup to pull in the cmake code from the https://github.com/quinoacomputing/cmake-modules repository, in quinoa as a subtree, in the TPL repository as a submodule.

Pull changes from the cmake repository

To pull in the latest updates from the cmake repository (from its branch master) into the quinoa repository, do

git fetch cmake
git subtree pull --prefix=cmake cmake master --squash

Push changes to the cmake repository

If you have committed changes in the quinoa repository under the cmake subtree directory cmake, here is how you push the changes upstream to the cmake git repository.

  • Verify the changes you are about to push by examining the diff output:
git diff cmake/master master:cmake
  • If you are happy with the diff, push the changes upstream (to the master branch of the cmake repository):
git subtree push --prefix=cmake cmake master

Useful links on git subtree

Git submodule for incorporating TPLs

To ease and automate building the third-party libraries that are not always available on a system, we maintain a git repository that consists of a list of git submodules pointing to the libraries. Check the Build section of the main page for a list of libraries that may be installed by your operating system's package manager, which then can accelerate and ease installing the TPLs.

When you do a clone of quinoa

git clone --recurse-submodules https://github.com/quinoacomputing/quinoa.git

all submodules are cloned and initialized. This is probably the simplest way to clone the code with all dependencies, however it is not necessarily the quickest to work with, since git operations will take longer when submodules are present and most developers will only need the submodules once (to build and install the TPLs).

Another way of cloning (and working with) quinoa is to not clone the submodules, but work with the git repositories for quinoa, cmake, and the TPLs separately. Since the cmake code and the TPLs are much less frequently updated compared to the code in quinoa, the TPLs can be cloned, built, and forgotten about. This approach, however, will require the specification of the -DTPL_DIR=<path-to-installed-tpls> command line argument to quinoa's cmake command. This way, the quinoa repository, which still has the cmake directory as a git subtree, is kept minimal, and thus git operations, e.g., switching branches, stashing, are quicker. This can be especially useful on large HPC machines where the large parallel file systems are shared among many users who may be doing heavy I/O. Another option is to deinitialize the submodules once not needed if they were recursively cloned and initialized.

Add a new TPL as a new submodule

This section walks through on adding a new TPL as a git submodule. We add a new library Sol2, which is header-only (no build required, only a copy of the header files) and depends on Lua.

First, we tell the quinoa-tpl repository about the the source repository of Sol2 as a new git submodule:

git clone https://github.com/quinoacomputing/quinoa-tpl.git
cd quinoa-tpl
git submodule add https://github.com/ThePhD/sol2.git src/sol2
cd src/sol2 && git checkout v3.2.0 && cd -
git add src/sol2

Then we extend quinoa-tpl/CMakeLists.txt. The best is to find a library similar to the one being added and add the relevant sections. Since Sol2 is a header-only library, we just find everything brigand-related (brigand is also a header-only library) and end up with the following diff:

$ git diff CMakeLists.txt
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 32e715d..595bed9 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -92,6 +92,7 @@ set(ENABLE_BACKWARDCPP ON CACHE BOOL "Enable BackwardCpp")
 set(ENABLE_HIGHWAYHASH ON CACHE BOOL "Enable HighwayHash")
 set(ENABLE_OMEGAH OFF CACHE BOOL "Enable Omega_H")
 set(ENABLE_BRIGAND ON CACHE BOOL "Enable Brigand")
+set(ENABLE_SOL2 ON CACHE BOOL "Enable Sol2")
 set(ENABLE_DOXYGEN ON CACHE BOOL "Enable Doxygen")
 set(ENABLE_MCSS ON CACHE BOOL "Enable m.css")

@@ -124,6 +125,7 @@ if (UNITTEST_ONLY OR INCITER_ONLY OR RNGTEST_ONLY OR MESHCONV_ONLY OR
   set(ENABLE_HIGHWAYHASH OFF)
   set(ENABLE_OMEGAH OFF)
   set(ENABLE_BRIGAND OFF)
+  set(ENABLE_SOL2 OFF)
   set(ENABLE_DOXYGEN OFF)
   set(ENABLE_MCSS OFF)
   set(REQUESTED_EXECUTABLES)
@@ -157,6 +159,7 @@ if (UNITTEST_ONLY OR INCITER_ONLY OR RNGTEST_ONLY OR MESHCONV_ONLY OR
     set(ENABLE_HDF5 ON)
     set(ENABLE_HIGHWAYHASH ON)
     set(ENABLE_BRIGAND ON)
+    set(ENABLE_SOL2 ON)
     set(ENABLE_PEGTL ON)
     set(ENABLE_LAPACK ON)
     set(ENABLE_BOOST ON)
@@ -724,6 +727,17 @@ if (ENABLE_BRIGAND)
   endif()
 endif()

+# Sol2
+if (ENABLE_SOL2)
+  find_package(Lua)
+  find_package(Sol2)
+  if(SOL2_FOUND)
+    set(sol2 "")
+  else()
+    set(sol2 "sol2")
+  endif()
+endif()
+
 # Doxygen
 if (ENABLE_DOXYGEN)
   find_package(Doxygen 1.8.15)
@@ -758,7 +772,7 @@ get_compiler_flags()
 set(tpls2build ${charm} ${hdf5} ${netcdf} ${boost} ${pstreams}
 ${pugixml} ${pegtl} ${random123} ${rngsse2} ${lapack} ${aec} ${h5part}
 ${testu01} ${trilinos} ${tut} ${numdiff} ${root} ${backwardcpp} ${highwayhash}
-${omega_h} ${brigand} ${doxygen} ${mcss})
+${omega_h} ${brigand} ${sol2} ${doxygen} ${mcss})

 list(LENGTH tpls2build ntpl)
 list(SORT tpls2build)
@@ -1333,6 +1347,23 @@ if (brigand)
   )
 endif()

+#### Sol2 ######################################################################
+# https://github.com/ThePhD/sol2
+# Header only, only if not found
+if (sol2)
+  ExternalProject_Add(
+    sol2
+    PREFIX sol2
+    # Header-only, copy include dir over
+    CONFIGURE_COMMAND ${CMAKE_COMMAND} -E copy_directory
+                      ${PROJECT_SOURCE_DIR}/src/sol2/include/sol
+                      ${CMAKE_INSTALL_PREFIX}/include/sol
+    BUILD_COMMAND ""
+    INSTALL_COMMAND ""
+    DOWNLOAD_COMMAND ""
+  )
+endif()
+
 #### Doxygen, documentation generation #########################################
 # https://doxygen.org
 if (doxygen)

We also add cmake code to find Sol2, as a new file cmake/FindSol2.cmake, since we do not expect that to be packaged by the OS.

################################################################################
#
# \file      FindSol2.cmake
# \copyright 2012-2015 J. Bakosi,
#            2016-2018 Los Alamos National Security, LLC.,
#            2019 Triad National Security, LLC.
#            All rights reserved. See the LICENSE file for details.
# \brief     Find the Sol2 Lua C++ binding library
#
################################################################################

# Sol2: https://github.com/ThePhD/sol2
#
#  SOL2_FOUND - System has Sol2
#  SOL2_INCLUDE_DIRS - The Sol2 include directory
#
#  Set the SOL2_ROOT cmake variable or shell environment variable before
#  calling find_package to a path to add an additional search path, e.g.,
#
#  Usage:
#
#  set(SOL2_ROOT "/path/to/custom/brigand") # prefer over system
#  find_package(Sol2)
#  include_directories(${SOL2_INCLUDE_DIRS})

# If already in cache, be silent
if(SOL2_INCLUDE_DIRS)
  set (SOL2_FIND_QUIETLY TRUE)
endif()

find_path(SOL2_INCLUDE_DIR NAMES sol.hpp
                           HINTS ${SOL2_ROOT}/include
                                 $ENV{SOL2_ROOT}/include
                           PATH_SUFFIXES sol)

set(SOL2_INCLUDE_DIRS ${SOL2_INCLUDE_DIR})

# Handle the QUIETLY and REQUIRED arguments and set SOL2_FOUND to TRUE if
# all listed variables are TRUE.
INCLUDE(FindPackageHandleStandardArgs)
FIND_PACKAGE_HANDLE_STANDARD_ARGS(Sol2 REQUIRED_VARS SOL2_INCLUDE_DIRS)

MARK_AS_ADVANCED(SOL2_INCLUDE_DIRS)

We can now test if the TPL build works:

cd quinoa-tpl && mdkir build && cd build
cmake ..

Example cmake screen output:

...
-- Found Lua: /usr/lib/x86_64-linux-gnu/liblua5.3.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "5.3.3")
-- Could NOT find Sol2 (missing: SOL2_INCLUDE_DIRS)
...

As expected, Lua is there on the system, but we need to supply Sol2. Now we build sol2, and see if it is found:

make sol2
cmake .

Example cmake screen output:

...
-- Found Lua: /usr/lib/x86_64-linux-gnu/liblua5.3.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "5.3.3")
-- Found Sol2: [...]/quinoa-tpl/install/clang-x86_64/include/sol
...

Since this works fine, we commit the changes to the quinoa-tpl repository:

git add CMakeLists.txt
git status
git commit -m"Add lua and sol2"

Example git status output from above:

On branch master
Your branch is up to date with 'origin/master'.

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   .gitmodules
        modified:   CMakeLists.txt
        new file:   src/sol2

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)

        modified:   cmake (untracked content)
...

We also update cmake/TPLs.cmake in the cmake-modules repository to include and run cmake code that finds Lua and Sol2 when quinoa is configured.

First, clone cmake-modules:

git clone https://github.com/quinoacomputing/cmake-modules.git
cd cmake-modules

Next, augment TPLs.cmake in the cmake-module repository, which runs when quinoa is configured:

$ git diff TPLs.cmake
diff --git a/cmake/TPLs.cmake b/cmake/TPLs.cmake
index a7ca35a93..f4064ea2c 100644
--- a/cmake/TPLs.cmake
+++ b/cmake/TPLs.cmake
@@ -156,6 +156,11 @@ find_package(HighwayHash)
 set(BRIGAND_ROOT ${TPL_DIR}) # prefer ours
 find_package(Brigand)

+#### Configure Sol2
+set(SOL2_ROOT ${TPL_DIR}) # prefer ours
+find_package(Lua)
+find_package(Sol2)
+
 message(STATUS "------------------------------------------")

 # Function to print a list of missing library names
@@ -197,11 +202,12 @@ endif()

 if (CHARM_FOUND AND SEACASExodus_FOUND AND EXODIFF_FOUND AND
     Zoltan2_FOUND AND HDF5_FOUND AND BRIGAND_FOUND AND PEGTL_FOUND AND
-    (MKL_FOUND OR LAPACKE_FOUND) AND Boost_FOUND AND HIGHWAYHASH_FOUND)
+    (MKL_FOUND OR LAPACKE_FOUND) AND Boost_FOUND AND HIGHWAYHASH_FOUND AND
+    LUA_FOUND AND SOL2_FOUND)
   set(ENABLE_INCITER "true")
   set(INCITER_EXECUTABLE inciter)
 else()
-  PrintMissing(inciter "CHARM_FOUND;SEACASExodus_FOUND;EXODIFF_FOUND;Zoltan2_FOUND;HDF5_FOUND;BRIGAND_FOUND;PEGTL_FOUND;MKL_FOUND;LAPACKE_FOUND;Boost_FOUND")
+  PrintMissing(inciter "CHARM_FOUND;SEACASExodus_FOUND;EXODIFF_FOUND;Zoltan2_FOUND;HDF5_FOUND;BRIGAND_FOUND;PEGTL_FOUND;MKL_FOUND;LAPACKE_FOUND;Boost_FOUND;LUA_FOUND;SOL2_FOUND")
 endif()

 if (CHARM_FOUND AND TESTU01_FOUND AND BRIGAND_FOUND AND PEGTL_FOUND AND

Now we commit the changes to cmake-modules:

cd cmake-modules
mv quinoa-tpl/cmake/FindSol2.cmake .
git add FindSol2.cmake TPLs.cmake
git commit -m"Add cmake code so quinoa finds Lua and Sol2"
git push
cd quinoa-tpl/cmake
git pull

Next we commit the change in the TPL repo to reflect the updated cmake submodule:

cd quinoa-tpl
git add cmake
git commit -m"Pull in latest cmake"
git push

Finally, we pull the changes from both the cmake subtree (cmake-modules repository) and the TPLs (quinoa-tpl) into the quinoa repository.

git clone https://github.com/quinoacomputing/quinoa.git
cd quinoa
git submodule init && git submodule update --recursive && cd external && git submodule init && git submodule update --recursive && cd .. && git submodule status --recursive
cd external && git checkout master && git pull && cd -
cd cmake && git checkout master && git pull && cd -
git submodule update --init external/src/sol2
git commit -am"Pull in latest external"
git fetch cmake
git subtree pull --prefix=cmake cmake master --squash