DIY 3.0 data-parallel out-of-core C++ library

- Main Page
- Modules
- Namespaces
- Classes
- Files
- Examples

All Classes Namespaces Functions Typedefs Groups Pages
Tutorial

## Getting started

The following is an example of the steps needed to initialize DIY.

### Block

The block is the basic unit of everything (data, decomposition, communication) in DIY. Use it to define your data model and any state associated with the data that will be needed to accompany the data throughout its lifetime. In addition to the data in the block, it should define functions to `create` and `destroy` the block that DIY can call. If the blocks are intended to be moved in and out of core, then the block must also define `save` and `load` functions.

#include <diy/master.hpp>
struct Block
{
static void* create() { return new Block; }
static void destroy(void* b) { delete static_cast<Block*>(b); }
static void save(const void* b, diy::BinaryBuffer& bb) { diy::save(bb, *static_cast<const Block*>(b)); }
static void load(void* b, diy::BinaryBuffer& bb) { diy::load(bb, *static_cast<Block*>(b)); }
// other functions and data members
}

### Master

diy::Master owns and manages the blocks. To set up a `Master` object, first define the MPI communicator and the file storage object (if blocks will be moved in and out of core), which you pass to the `Master` constructor. The `Master` manages loading/saving blocks, executing their callback functions, and exchanging data between them.

int main(int argc, char** argv)
{
...
diy::mpi::communicator world(comm);
diy::FileStorage storage("./DIY.XXXXXX");
diy::Master master(world,
num_threads,
mem_blocks,
&Block::create,
&Block::destroy,
&storage,
&Block::save,
&Block::load);
...
}

Some of the arguments to the constructor are optional. If all blocks are to remain in memory, there is a shorter form of the `Master` constructor that can be used, since there is no need to specify most of the arguments because they relate to block loading/unloading.

diy::Master master(world,
num_threads);

### Assigner

diy::Assigner is an auxiliary object that determines what blocks lives on what MPI process. Blocks can be assigned to processes contiguously or in round-robin fashion:

diy::ContiguousAssigner assigner(world.size(), // total number of MPI ranks
nblocks); // total number of blocks in global domain
// --- or ---
diy::RoundRobinAssigner assigner(world.size(),
nblocks);

### Decomposition

Any custom decomposition can be formed by assigning links (communication neighborhoods of blocks) manually. However, for a regular grid of blocks, DIY provides a regular decomposition of blocks with either continuous (floating-point extents that share common boundaries) or discrete (integer extents that may or may not overlap) bounds.

typedef diy::ContinuousBounds Bounds;
typedef diy::RegularContinuousLink RCLink;
// --- or ---
typedef diy::DiscreteBounds Bounds;
typedef diy::RegularGridLink RGLink;

`RegularDecomposer` helps decompose a domain into a regular grid of blocks. It's initialized with the dimension of the domain, its extents, and the number of blocks used in the decomposition.

diy::RegularDecomposer<Bounds> decomposer(dim,
domain,
nblocks);

Its member function `decompose` performs the actual decomposition. Besides the local MPI rank and an instance of `Assigner`, it takes a callback responsible for creating the block and adding it to a `Master`. In C++11, it's convenient to use a lambda function for this purpose.

decomposer.decompose(rank, assigner,
[&](int gid, // block global id
const Bounds& core, // block bounds without any ghost added
const Bounds& bounds, // block bounds including any ghost region added
const Bounds& domain, // global data bounds
const RCLink& link) // neighborhood
{
Block* b = new Block; // possibly use custom initialization
RGLink* l = new RGLink(link);
int lid = master.add(gid, b, l); // add block to the master (mandatory)
// process any additional args here, load the data, etc.
});

A shorter form is provided, if you only want to add the blocks to `Master`, without any additional processing.

decomposer.decompose(rank,
assigner,
master);

### Combined code

Here is one version of each of the above options combined into a complete program. This example uses the short form of blocks:

#include <diy/decomposition.hpp>
#include <diy/assigner.hpp>
#include <diy/master.hpp>
typedef diy::DiscreteBounds Bounds;
typedef diy::RegularGridLink RGLink;
struct Block
{
Block() {}
static void* create() { return new Block; }
static void destroy(void* b) { delete static_cast<Block*>(b); }
};
int main(int argc, char* argv[])
{
diy::mpi::environment env(argc, argv); // diy's version of MPI_Init
diy::mpi::communicator world; // diy's version of MPI communicator
int dim = 3; // dimensionality
int size = 8; // total number of MPI ranks
int nblocks = 32; // total number of blocks in global domain
diy::ContiguousAssigner assigner(size, nblocks);
Bounds domain; // global data size
diy::Master master(world,
1, // 1 thread in this example
-1, // all blocks in memory in this example
&Block::create, // block create function
&Block::destroy); // block destroy function
diy::RegularDecomposer<Bounds> decomposer(dim,
domain, // global domain size
nblocks);
decomposer.decompose(world.rank(), // MPI rank of this process
assigner, // assigner object
master); // master object
}

Below is one more complete example, using a lambda function to initialize the block:

#include <diy/decomposition.hpp>
#include <diy/assigner.hpp>
#include <diy/master.hpp>
typedef diy::DiscreteBounds Bounds;
typedef diy::RegularGridLink RGLink;
struct Block
{
Block() {}
static void* create() { return new Block; }
static void destroy(void* b) { delete static_cast<Block*>(b); }
};
int main(int argc, char* argv[])
{
diy::mpi::environment env(argc, argv); // diy's version of MPI_Init
diy::mpi::communicator world; // diy's version of MPI communicator
int dim = 3; // dimensionality
int size = 8; // total number of MPI ranks
int nblocks = 32; // total number of blocks in global domain
diy::ContiguousAssigner assigner(size, nblocks);
Bounds domain; // global data size
diy::Master master(world,
1, // 1 thread in this example
-1, // all blocks in memory in this example
&Block::create, // block create function
&Block::destroy); // block destroy function
diy::RegularDecomposer<Bounds> decomposer(dim,
domain, // global domain size
nblocks);
decomposer.decompose(world.rank(), // MPI rank of this process
assigner, // assigner object
[&](int gid, // block global id
const Bounds& core, // block bounds without any ghost added
const Bounds& bounds, // block bounds including any ghost region added
const Bounds& domain, // global data bounds
const RCLink& link) // neighborhood
{
Block* b = new Block; // possibly use custom initialization
RGLink* l = new RGLink(link);
int lid = master.add(gid, b, l); // add block to the master (mandatory)
});
}

diy::Master API has a separate documentation page. The `RegularDecomposer` is documented in the Decomposition page.