IO

Detailed Description

Block I/O: DIY writes and reads blocks to and from storage in either collective or independent mode. The collective mode, which is the default, creates one file. Independent mode creates one file per process. These features should not be confused with the temporary block I/O that DIY does in the course of shuffling blocks in and out of core while executing the block foreach functions. This happens behind the scenes and is not the subject of the I/O module.

Collective I/O

• diy::io::write_blocks()
• diy::io::read_blocks()

Independent I/O

• diy::io::split::write_blocks()
• diy::io::split::read_blocks()

An example (of collective I/O) is below:

int mem_blocks = -1;
int num_threads = 1;
diy::mpi::communicator    world(comm);
diy::FileStorage          storage("./DIY.XXXXXX");
diy::Master               master(world,
                                 num_threads,
                                 mem_blocks);


// --- write ---

// the total number of blocks, tot_blocks, is set by user
diy::ContiguousAssigner assigner(world.size(), tot_blocks);
// the block_save function is unnecessary if it was defined as part of master
diy::io::write_blocks(filename, world, master, &block_save);


// --- read ---

// the number of blocks is -1 because it is determined by read_blocks()
diy::ContiguousAssigner assigner(world.size(), -1);
// the block_load function is unnecessary if it was defined as part of master
diy::io::read_blocks(filename, world, *assigner, master, &block_load);

In addition, the examples/io directory illustrates BOV (brick of values) and NumPy I/O, which is built on top of MPI-IO and MPI sub-array types. The test-io.cpp example tests the readers and writers for BOV and NumPy.

Functions
void	diy::io::write_blocks (const std::string &outfilename, const mpi::communicator &comm, Master &master, const MemoryBuffer &extra=MemoryBuffer(), Master::SaveBlock save=0)
	Write blocks to storage collectively in one shared file. More...
void	diy::io::read_blocks (const std::string &infilename, const mpi::communicator &comm, Assigner &assigner, Master &master, MemoryBuffer &extra, Master::LoadBlock load=0)
	Read blocks from storage collectively from one shared file. More...
void	diy::io::split::write_blocks (const std::string &outfilename, const mpi::communicator &comm, Master &master, const MemoryBuffer &extra=MemoryBuffer(), Master::SaveBlock save=0)
	Write blocks to storage independently in one file per process. More...
void	diy::io::split::read_blocks (const std::string &infilename, const mpi::communicator &comm, Assigner &assigner, Master &master, MemoryBuffer &extra, Master::LoadBlock load=0)
	Read blocks from storage independently from one file per process. More...

Function Documentation

void diy::io::read_blocks ( const std::string &  infilename, const mpi::communicator &  comm, Assigner &  assigner, Master &  master, MemoryBuffer &  extra, Master::LoadBlock  load = 0  )

inline

Read blocks from storage collectively from one shared file.

Parameters

infilename	input file name
comm	communicator
assigner	assigner object
master	master object
extra	user-defined metadata in file header
load	load block function in case different than or unefined in the master

Examples:

simple/read-blocks.cpp.

void diy::io::split::read_blocks ( const std::string &  infilename, const mpi::communicator &  comm, Assigner &  assigner, Master &  master, MemoryBuffer &  extra, Master::LoadBlock  load = 0  )

inline

Read blocks from storage independently from one file per process.

Parameters

infilename	input file name
comm	communicator
assigner	assigner object
master	master object
extra	user-defined metadata in file header
load	block load function in case different than or undefined in master

void diy::io::write_blocks ( const std::string &  outfilename, const mpi::communicator &  comm, Master &  master, const MemoryBuffer &  extra = MemoryBuffer(), Master::SaveBlock  save = 0  )

inline

Write blocks to storage collectively in one shared file.

Parameters

outfilename	output file name
comm	communicator
master	master object
extra	user-defined metadata for file header; meaningful only on rank == 0
save	block save function in case different than or undefined in the master

Examples:

simple/simple.cpp.

void diy::io::split::write_blocks ( const std::string &  outfilename, const mpi::communicator &  comm, Master &  master, const MemoryBuffer &  extra = MemoryBuffer(), Master::SaveBlock  save = 0  )

inline

Write blocks to storage independently in one file per process.

Parameters

outfilename	output file name
comm	communicator
master	master object
extra	user-defined metadata for file header; meaningful only on rank == 0
save	block save function in case different than or undefined in master