DIY 3.0 data-parallel out-of-core C++ library

- Main Page
- Modules
- Namespaces
- Classes
- Files
- Examples

All Classes Namespaces Functions Typedefs Groups Pages
Functions
IO

## Detailed Description

Block I/O: DIY writes and reads blocks to and from storage in either collective or independent mode. The collective mode, which is the default, creates one file. Independent mode creates one file per process. These features should not be confused with the temporary block I/O that DIY does in the course of shuffling blocks in and out of core while executing the block `foreach` functions. This happens behind the scenes and is not the subject of the I/O module.

## Collective I/O

- `diy::io::write_blocks()`
- `diy::io::read_blocks()`

## Independent I/O

- `diy::io::split::write_blocks()`
- `diy::io::split::read_blocks()`

An example (of collective I/O) is below:

int mem_blocks = -1;
int num_threads = 1;
diy::mpi::communicator world(comm);
diy::FileStorage storage("./DIY.XXXXXX");
diy::Master master(world,
num_threads,
mem_blocks);
// --- write ---
// the total number of blocks, tot_blocks, is set by user
diy::ContiguousAssigner assigner(world.size(), tot_blocks);
// the block_save function is unnecessary if it was defined as part of master
diy::io::write_blocks(filename, world, master, &block_save);
// --- read ---
// the number of blocks is -1 because it is determined by read_blocks()
diy::ContiguousAssigner assigner(world.size(), -1);
// the block_load function is unnecessary if it was defined as part of master
diy::io::read_blocks(filename, world, *assigner, master, &block_load);

In addition, the examples/io directory illustrates BOV (brick of values) and NumPy I/O, which is built on top of MPI-IO and MPI sub-array types. The test-io.cpp example tests the readers and writers for BOV and NumPy.

## Functions
void | diy::io::write_blocks (const std::string &outfilename, const mpi::communicator &comm, Master &master, const MemoryBuffer &extra=MemoryBuffer(), Master::SaveBlock save=0)
| Write blocks to storage collectively in one shared file. More...
void | diy::io::read_blocks (const std::string &infilename, const mpi::communicator &comm, Assigner &assigner, Master &master, MemoryBuffer &extra, Master::LoadBlock load=0)
| Read blocks from storage collectively from one shared file. More...
void | diy::io::split::write_blocks (const std::string &outfilename, const mpi::communicator &comm, Master &master, const MemoryBuffer &extra=MemoryBuffer(), Master::SaveBlock save=0)
| Write blocks to storage independently in one file per process. More...
void | diy::io::split::read_blocks (const std::string &infilename, const mpi::communicator &comm, Assigner &assigner, Master &master, MemoryBuffer &extra, Master::LoadBlock load=0)
| Read blocks from storage independently from one file per process. More...

## Function Documentation

void diy::io::read_blocks | ( | const std::string & | infilename,
|  | const mpi::communicator & | comm,
|  | Assigner & | assigner,
|  | Master & | master,
|  | MemoryBuffer & | extra,
|  | Master::LoadBlock | load = `0`
| ) |  |
| inline

Read blocks from storage collectively from one shared file.

Parameters

  infilename | input file name
  comm | communicator
  assigner | assigner object
  master | master object
  extra | user-defined metadata in file header
  load | load block function in case different than or unefined in the master

Examples:
  simple/read-blocks.cpp.

void diy::io::split::read_blocks | ( | const std::string & | infilename,
|  | const mpi::communicator & | comm,
|  | Assigner & | assigner,
|  | Master & | master,
|  | MemoryBuffer & | extra,
|  | Master::LoadBlock | load = `0`
| ) |  |
| inline

Read blocks from storage independently from one file per process.

Parameters

  infilename | input file name
  comm | communicator
  assigner | assigner object
  master | master object
  extra | user-defined metadata in file header
  load | block load function in case different than or undefined in master

void diy::io::write_blocks | ( | const std::string & | outfilename,
|  | const mpi::communicator & | comm,
|  | Master & | master,
|  | const MemoryBuffer & | extra = `MemoryBuffer()`,
|  | Master::SaveBlock | save = `0`
| ) |  |
| inline

Write blocks to storage collectively in one shared file.

Parameters

  outfilename | output file name
  comm | communicator
  master | master object
  extra | user-defined metadata for file header; meaningful only on rank == 0
  save | block save function in case different than or undefined in the master

Examples:
  simple/simple.cpp.

void diy::io::split::write_blocks | ( | const std::string & | outfilename,
|  | const mpi::communicator & | comm,
|  | Master & | master,
|  | const MemoryBuffer & | extra = `MemoryBuffer()`,
|  | Master::SaveBlock | save = `0`
| ) |  |
| inline

Write blocks to storage independently in one file per process.

Parameters

  outfilename | output file name
  comm | communicator
  master | master object
  extra | user-defined metadata for file header; meaningful only on rank == 0
  save | block save function in case different than or undefined in master