DIY 3.0 data-parallel out-of-core C++ library

- Main Page
- Modules
- Namespaces
- Classes
- Files
- Examples

All Classes Namespaces Functions Typedefs Groups Pages
Functions
Initialization

## Detailed Description

## Getting started

The following is an example of the steps needed to initialize diy:

### Start with a definition of a block.

The block is the basic unit of everything (data, decomposition, communication) in diy. Use it to define your data model and any state associated with the data that will be needed to accompany the data throughout its lifetime. In addition to the data in the block, it should define functions to `create` and `destroy` the block that diy can call. If the blocks are intended to be moved in and out of core, then the block must also define `save` and `load` functions.

There are 2 common ways to set up the block.

- The "short form" is used when blocks do not need any arguments when created. Here is an example:

#include <diy/master.hpp>
struct Block
{
static void* create() { return new Block; }
static void destroy(void* b) { delete static_cast<Block*>(b); }
static void save(const void* b, diy::BinaryBuffer& bb)
{ diy::save(bb, *static_cast<const Block*>(b)); }
static void load(void* b, diy::BinaryBuffer& bb)
{ diy::load(bb, *static_cast<Block*>(b)); }
// other functions and data members
}

- The "long form" is used when custom arguments are needed to initialize blocks upon creation. This saves one I/O cycle if blocks are moved in and out of core and also provides useful information to the block such as the bounds. A second object called `AddBlock` is a functor that gets and stores any custom arguments and overloads the function call with access to the custom arguments:

#include <diy/master.hpp>
typedef diy::ContinuousBounds Bounds;
typedef diy::RegularContinuousLink RCLink;
struct Block
{
static void* create() { return new Block; }
static void destroy(void* b) { delete static_cast<Block*>(b); }
static void save(const void* b, diy::BinaryBuffer& bb)
{ diy::save(bb, *static_cast<const Block*>(b)); }
static void load(void* b, diy::BinaryBuffer& bb)
{ diy::load(bb, *static_cast<Block*>(b)); }
// other functions and data members
};
struct AddBlock
{
AddBlock(diy::Master& master_) : master(master_) {} // additional args as needed
// this is the function that is needed for diy::decompose
void operator()(int gid, // block global id
const Bounds& core, // block bounds without any ghost added
const Bounds& bounds, // block bounds including any ghost region added
const Bounds& domain, // global data bounds
const RCLink& link) // neighborhood
const
{
Block* b = new Block();
RGLink* l = new RGLink(link);
diy::Master& m = const_cast<diy::Master&>(master);
int lid = m.add(gid, b, l); // add block to the master (mandatory)
// process any additional args here, using them to initialize the block
}
diy::Master& master;
// store additional args here
};

### Create the master object.

Define your global data bounds, the MPI communicator, and the file storage object (if blocks will be moved in and out of core). Then define the diy `Master` object. The `Master` manages loading/saving blocks, executing their callback functions, and exchanging data between them. If the block long form (with `AddBlock` functor) is used, define the `AddBlock` object as well.

int main(int argc, char** argv)
{
...
diy::Bounds domain;
diy::mpi::communicator world(comm);
diy::FileStorage storage("./DIY.XXXXXX");
diy::Master master(world,
num_threads,
mem_blocks,
&Block::create,
&Block::destroy,
&storage,
&Block::save,
&Block::load);
// if AddBlock is used (block long form), then:
AddBlock addblock(master) // could have extra args
...
}

When all blocks will remain in memory, there is a shorter form of the `Master` constructor that can be used. In this case there is no need to specify most of the arguments because they relate to block loading/unloading.

diy::Master master(world,
num_threads);

### Assign blocks to processes.

Blocks can be assigned to processes contiguously or in round-robin fashion:

diy::ContiguousAssigner assigner(world.size(), // total number of MPI ranks
nblocks); // total number of blocks in global domain
// --- or ---
diy::RoundRobinAssigner assigner(world.size(),
nblocks);

### Decompose the domain.

Any custom decomposition can be formed by assigning links (communication neighborhoods of blocks) manually. However, for a regular grid of blocks, diy provides a regular decomposition of blocks with either continuous (floating-point extents that share common boundaries) or discrete (integer extents that may or may not overlap) bounds.

typedef diy::ContinuousBounds Bounds;
typedef diy::RegularContinuousLink RCLink;
// --- or ---
typedef diy::DiscreteBounds Bounds;
typedef diy::RegularGridLink RGLink;

Diy offers two versions of a `decompose` function, depending on whether the short or long form of the block is used. Recall that the long form has an `AddBlock` functor, while the short form does not. In the short form, one of the arguments to `decompose` is `master`, and in the long form, provide `addblock` instead:

// in the short form, provide the master object in lieu of an AddBlock object
diy::decompose(dim,
rank,
domain,
assigner,
master);
// --- or ---
// in the long form, provide the AddBlock object
diy::decompose(dim,
rank,
domain,
assigner,
addblock);

Strictly speaking, the second (long form) decomposition does not require an entire `AddBlock` object, as long as the block create function has the same signature as `AddBlock`. That pattern is:

void create(int gid,
const Bounds& core,
const Bounds& bounds,
const Bounds& domain,
const diy::Link& link);
...
diy::decompose(dim,
rank,
domain,
assigner,
create);

`decompose` is actually a convenience function that creates a `RegularDecomposer` object, calls its `decompose` method, and then destroys the `RegularDecomposer`. The user may want persistent access to `RegularDecomposer` because it offers useful information about the decomposition (block bounds, numbers of blocks in each dimension, and so forth.). In that case, we suggest defining `RegularDecomposer` and calling its `decompose` function yourself:

diy::RegularDecomposer<Bounds> decomposer(dim,
domain,
nblocks);
// in the short form, use diy's AddBlock object
decomposer.decompose(rank,
assigner,
diy::detail::AddBlock<Bounds>(&master));
// --- or ---
// in the long form, provide your own AddBlock object
decomposer.decompose(rank,
assigner,
addblock);

### Put it all together.

Here is one version of each of the above options combined into a complete program. This example uses the short form of blocks:

#include <diy/decomposition.hpp>
#include <diy/assigner.hpp>
#include <diy/master.hpp>
typedef diy::DiscreteBounds Bounds;
typedef diy::RegularGridLink RGLink;
struct Block
{
Block() {}
static void* create() { return new Block; }
static void destroy(void* b) { delete static_cast<Block*>(b); }
};
int main(int argc, char* argv[])
{
diy::mpi::environment env(argc, argv); // diy's version of MPI_Init
diy::mpi::communicator world; // diy's version of MPI communicator
int dim = 3; // dimensionality
int size = 8; // total number of MPI ranks
int nblocks = 32; // total number of blocks in global domain
diy::ContiguousAssigner assigner(size, nblocks);
Bounds domain; // global data size
diy::Master master(world,
1, // 1 thread in this example
-1, // all blocks in memory in this example
&Block::create, // block create function
&Block::destroy); // block destroy function
diy::decompose(dim,
world.rank(), // MPI rank of this process
domain, // global domain size
assigner, // assigner object
master); // master object
}

Below is one more complete example, this time using the long form of the block:

#include <diy/decomposition.hpp>
#include <diy/assigner.hpp>
#include <diy/master.hpp>
typedef diy::DiscreteBounds Bounds;
typedef diy::RegularGridLink RGLink;
struct Block
{
Block() {}
static void* create() { return new Block; }
static void destroy(void* b) { delete static_cast<Block*>(b); }
};
struct AddBlock
{
AddBlock(diy::Master& master_) : master(master_) {}
// this is the function that is needed for diy::decompose
void operator()(int gid, // block global id
const Bounds& core, // block bounds without any ghost
const Bounds& bounds, // block bounds including ghost region
const Bounds& domain, // global data bounds
const RGLink& link) // communication neighborhood
const
{
Block* b = new Block();
RGLink* l = new RGLink(link);
diy::Master& m = const_cast<diy::Master&>(master);
int lid = m.add(gid, b, l); // add block to the master (mandatory)
}
diy::Master& master;
};
int main(int argc, char* argv[])
{
diy::mpi::environment env(argc, argv); // diy's version of MPI_Init
diy::mpi::communicator world; // diy's version of MPI communicator
int dim = 3; // dimensionality
int size = 8; // total number of MPI ranks
int nblocks = 32; // total number of blocks in global domain
diy::ContiguousAssigner assigner(size, nblocks);
Bounds domain; // global data size
diy::Master master(world,
1, // 1 thread in this example
-1, // all blocks in memory in this example
&Block::create, // block create function
&Block::destroy); // block destroy function
AddBlock addblock(master); // object for adding new blocks to master
diy::decompose(dim,
world.rank(), // MPI rank of this process
domain, // global domain size
assigner, // assigner object
addblock); // add block object
}

Documentation about the `Master` object appears below. The `RegularDecomposer` is documented in the Decomposition page.

## Functions
| diy::Master::Master (mpi::communicator comm, int threads=1, int limit=-1, CreateBlock create=0, DestroyBlock destroy=0, ExternalStorage *storage=0, SaveBlock save=0, LoadBlock load=0, QueuePolicy *q_policy=new QueueSizePolicy(4096))
| The main DIY object. More...

## Function Documentation

diy::Master::Master | ( | mpi::communicator | comm,
|  | int | threads = `1`,
|  | int | limit = `-1`,
|  | CreateBlock | create = `0`,
|  | DestroyBlock | destroy = `0`,
|  | ExternalStorage * | storage = `0`,
|  | SaveBlock | save = `0`,
|  | LoadBlock | load = `0`,
|  | QueuePolicy * | q_policy = `new QueueSizePolicy(4096)`
| ) |  |
| inline

The main DIY object.

Helper functions specify how to: create an empty block, destroy a block (a function that's expected to upcast and delete), serialize a block

Parameters

  comm | communicator
  threads | number of threads DIY can use
  limit | number of blocks to store in memory
  create | block create function; master manages creation if create != 0
  destroy | block destroy function; master manages destruction if destroy != 0
  storage | storage object (path, method, etc.) for storing temporary blocks being shuffled in/out of core
  save | block save function; master manages saving if save != 0
  load | block load function; master manages loading if load != 0