# Lattice Boltzmann Method

Lattice Boltzmann methods (LBM) is a class of computational fluid dynamics (CFD) methods for fluid simulation

Feel free to create an issue for any questions, bugs, or general help.

You can also contact us at kitware@kitware.com.

## Cloning the Repository

Using a bash prompt

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~bash
$ mkdir LBM
$ cd LBM
# Pull the source, Note the period in the following line
$ git clone https://gitlab.kitware.com/LBM/lattice-boltzmann-solver.git .
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

## Build Environment

The code provided utilizes C++11 and you have installed [CUDA](https://developer.nvidia.com/cuda-downloads) on your machine.
- Minimum of CUDA 9.2
- Minimum of GCC 6
- Minimum of MSVC 2017

Our development environments generally consist of combinations of MSVC 2019 with CUDA 10.2 and GCC-6 with CUDA 9.2 and 10.2

Note to GCC-5 users, both CUDA 9.0 and 10.2 failed to build due to C++11 usage in the CUDA file, it is neccssary to upgrade to GCC-6 solves this issue. If you have multiple CUDA versions and/or GCC versions, it may be necessary to specify CMAKE and or CUDA configuration variables to properly build.

### Linux Environment

If you are building with VTK support please ensure you have OpenGL and X11 support installed.
To install OpenGL and x11 support :

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~bash
$ sudo apt-get install libgl1-mesa-dev
$ sudo apt-get install libxt-dev
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

## About our CMake

The provided project is a CMake project and will require CMake.
Go to the cmake website, <a href="https://cmake.org/download">https://cmake.org/download</a>, and download the appropriate distribution.
Ensure that the CMake bin is on your PATH and available in your cmd/bash shell.

The code is built as part of a CMake 'superbuild', meaning the build is broken into 2 phases, 
  1. A build target will be created that will download any dependent libraries and build them
  2. A build target will be created that will build the code specific to this project, using the newly built libraries.

By default, the LBM superbuild will download and build VTK 8.2, ITK 5.1, and Eigen 3.3.7
  
The build is also out of source, meaning the code base is seperated from the build files.
This means you will need two folders, one for the source code and one for the build files.
Throughout this ReadMe, the build directory will be refered to as `<build_dir>`.
The build directory is the disk location in which you originally run CMake, or specify via the CMake GUI.

### CMake Configuration Options

Run CMake on the source from your <build_dir> directory.
To change cmake options (ex. enable a CMake switch), you can do this several different ways.
In the following examples, I will use the CMake flag for Enabling the LBM_VTK_OFFSCREEN option.<br>
NOTE: ON is interchangable with YES, and OFF is interchangable with NO

  * Via Command Line : Add -DLBM_VTK_OFFSCREEN=ON
  * Via CMake GUI : Click to check the value field box associated with the LBM_VTK_OFFSCREEN line to change it
  * Via ccmake : Use arrows to highlight the BM_VTK_OFFSCREEN option and usethe spacebar to toggle the option state (ON or OFF)
  
To learn more about CMake visit the [Running CMake Help Page](https://cmake.org/runningcmake/)<br>

If you are using the CMake GUI, it is suggested to enable Grouped and Advanced in the CMake GUI.<br>
If you are using CMake via the command like, it is suggested to run CMake and use the cmake-curses-gui if you would like to modify any configuration opions<br>

NOTE: Modifying the CMakeCache.txt does not impact your generated build files (Makefile or MSVC solution).
To modify any generated build files, you must edit CMake options using the cmake gui or ccmake and configure and generate new build files.

#### Headless VTK Configuration (Linux Users)

LBM utilizes VTK to generate plot images when running our verification driver which uses the X Server display on Linux.

If you are building on a headless machine that does not have an X Sever display (such as in a Docker container or ssh connection to a remote machine) and `LBM_VTK_EXTENSION` is enabled (which is on by default) you will need to enable the `LBM_VTK_OFFSCREEN` options which is `OFF` by default.

To render offscreen you will need to install OSMesa

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~bash
$ sudo apt-get install libosmesa-dev
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you don't want to have these plot files generated, you can also disable the `LBM_VTK_EXTENSION` option.

An example of running CMake with this option is provided below:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~bash
# This assumes you are still in your source directory from the steps above
# Create a separate build directory
$ cd ..
$ mkdir LBM_build
$ cd LBM_build
# Run CMake with offscreen rendering enabled
$ cmake -DLBM_VTK_OFFSCREEN=ON ../LBM
# Build the superbuild project
$ make -j4
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

#### Using prebuilt software packages

By default, the LBM superbuild will download and build VTK 8.2, ITK 5.1, and Eigen 3.3.7<br>
If you would like to use a prebuilt version of any of these libraries, you can do so when configuring CMake
Note, LBM is currently only compatible with the 8.x versions of VTK. 
We will need to upgrade our CMake to be compatible with 9+

To do this, set USE_SYSTEM_VTK, USE_SYSTEM_ITK or USE_SYSTEM_Eigen via the CMake GUI or cmake-curses-gui if you would like to use your own builds of these software packages.

With these items enabled configure CMake.
New variables will be added to CMake, for example, if USE_SYSTEM_VTK was enabled, a new VTK_DIR variable will be created.
Set location containing the VTKConfig.cmake file.
Generally, this contained in the `<vtk install>/lib/cmake/vtk-8.2` folder.

### Generating CMake

Once all options are configured as desired, you can generate your build files. 

## Building

All executables will be built to the `<build_dir>/install` directory.

### MSVC Users

  - Open the LatticeBoltzmannMethod.sln in your specified <build_dir> directory and build one or more configurations (Debug and/or Release)
  - Once built, You may now close this solution file.
  - A new LatticeBoltzmannMethod.sln will be built to your `<build_dir>/Innerbuild` directory, open it for development.
  - Note if you are using a prebuilt VTK, you will need to add `PATH=/path/to/VTK/install/bin` to your active project Debugging/Environment property

### Bash Users

  - Call make from your <build_dir> directory
  - Once built, there will be a new makefile your `<build_dir>/Innerbuild` directory
  - The makefile proviede in the Innerbuild directory is used to quickly build and test any modifications to the LBM code base

## Running

This repository provides two drivers for executing the LBM solver.

### LBMVerificationDriver

The provided verification driver compares the LBM solution to the analytic solution for cylinder and brick geometries.
The cylinder and brick geometries used are procedurally generated. The geometry parameters are specified (length, brick width and height or cylinder radius) in the input `*.cfg` file.

The verification driver runs and plots the results of the current simulation (computed) vs the baseline (previously saved computed values) and the analytical solution. The error is also calculated and plotted between the current simulation and the baseline and the current simulation and the analytical solution. When the LBMVerification executable is run from the command line, each example case (flow through a brick, flow through a channel) is executed and the results are compared as described above. The plots can be viewed as image files in the /verification folder. 

Example Usage: 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~bash
# *Nix/OSX users, you will need to add the <build_dir>/install/bin directory to your library path (ex. LD_LIBRARY_PATH)
# You can do this yourself, or by running a script we provide in your install directory
$ <build_dir>/install> source setup_LBM.sh
# The LBM executables will be placed in your install/bin directory
$ cd <build_dir>/install/bin/
# Run the LBM Verification Driver for one cylinder case (should a few minutes)
$ ./LBMVerificationDriver cylinder0_spacing=0.001
# To run all test cases, run without any inputs (should take 30m-1hr)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This driver takes a folder name as input, and will recursivly search it for all `*.cfg` files.
When no argument is provided, the driver will open and read the `install/bin/run.config` file, and use the path provided to the `verification_dir` parameter and run all the verification runs provided in the reppository.
The `verification_dir`, by default, will contain the path to your LBM `source/data/verification` directory.

An LBM verification run consistes of a folder containing an input `*.cfg` file along with several csv files containing the expected results from the LBM solver.

When executed, the driver will:
  - Create a new folder in its working directory (install/bin) named `verification_runs`
  - Create the same folder hierarchy found under the input (or default) search folder.
  - Write all results and data files associated with the execution and analysis of each run to its analagous folder under `verification_runs`

#### Inputs

A configuration file needs to be supplied.
A set of examples are provided [here](data/verification/).
In file comments describe each property and its use.

#### Outputs

At the root of the `verification_runs` directory will be an html file containing a summary table of our verification comparisons for each configuration run and how the results compare to associated the analytic solution.

##### LBM_Verification_Report

The generated HTML report provides a table for each of our cylinder and brick verification runs.

  - Run : The name of the configuration file for this row
  - Total Test Time : Includes I/O, setup, execution, result generation and reporting
  - LBM Compute Time : Time spent in executing our LBM implementation on the provided model
  - LBM Iterations : The number of iterations performed
  - Nodes : The number of nodes for this model
  - MLUPS : Million Lattice Updates Per Second achieved for this run
  - Velocity Baseline Analytic vs. Computed Analytic : Comparison of the analytic solution from the pulled version of the repository with any changes that may have been introduced by the user. Ensures that the analytic solution has not been corrupted.
  - Max Avg Velocity Error Fraction Baseline LBM vs. Computed LBM : The baseline values are the values from the LBM current git hash without any modifications. This comparison error compares the developers current code changes with the LBM solution from the pulled version of the repository. Allows the user to evaluate how changes to the algorithm affect the solution.
  - Average Velocity Error Fraction Computed LBM vs Computed Analytic : Comparison of the values generated by the developers LBM code base with the computed analytic solution. This is an up to date analysis of the LBM solution as it compares to the analytic solution. Allows the user to assess how algorithm changes affect the validation metrics. 

The error is calculated by comparing the two solutions for each of the cases stated above. The error is calculated at each location as the difference between the two cross sections (y or z axis at the given x). The average error is then calculated across the axis and a percent error is calculated by dividing by the baseline or analytic solution depending on the case. For errors less than or equal to 2% difference, the simulation is considered to be validated (Pass) and is color coded as green. Larger errors are considered as a (Fail) and color coded red.

At this time, we expect three failures associated with the following geometries for the validation schema (Computed LBM vs Computed Analytic).
  - brick 0_spacing=0.0006
  - brick 0_spacing=0.0008
  - brick 0_spacing=0.001

##### Individual Run Files

Each run will also write output files to a directory under the `verification_runs` folder.
These run folders will contain both csv and image files associated with certain views of the generated results

###### CSV Files

The cylinder and brick extend down the x axis.
We walk down the x-axis and provide the temperature, velocity, pressure, flow, and stress values along both the z and y axis cells at each x cell. We also proved the error between the baseline and computed values as compared to the analytic solution.

###### Images

We create images for from the data provided in the CSV files. 
An image is created for each step along the x axis.
This image will show :

  - Values from the analytic solution in black
  - Values from the baseline solution in blue
  - Values generated from the compiled code in red
  
These images provide a developer the ability to see how changes compare against code from their last commit, as well as the analytic solution.

A VTR file is generated as output.
To view the VTR file, use [ParaView.](https://www.paraview.org/)

  - Open ParaView
  - Open the vtr file
  - Apply a threshold to the data on the boundary labels between 0 and 100 and click apply
    - This should now show your geometry
  - You can then select the data you would like to view and continue to process it using the ParaView tools

### LBMITKDriver

The ITKDriver uses the Insight Toolkit (ITK) to load a file describing a geometry and a configuration file specifying the boundary conditions and fluid and simulation parameters. Our example is a nasal geometry with parameters for airflow through the nasal passages.

Example Usage: 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~bash
# *Nix users, you will need to add the lib directory to your library path/to/LBM/install/bin
# You can do this by running a script from your install directory
$ <build_dir>/install source setup_LBM.in
# executable will be placed in your install/bin directory
$ cd <build_dir>/install/bin/
# Run the LBM ITK Driver, which only runs 1 example nasal geometry
# This run should only take a few minutes
$ ./LBMITKDriver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This driver takes a folder name as input, and will recursivly search it for all `*.cfg` files.
When no argument is provided, the driver will open and read the `install/bin/run.config` file, and use the path provided to the `imagery_dir` parameter
This file, by default, will contain a path to your LBM `source/data/itk` directory.

An LBM ITK run consistes of a folder containing an input `*.cfg` file along with a *.nrrd* file containing the medical imagery.

When executed, the driver will:

  - Create a new folder in its working directory named `itk_runs`
  - Create the same folder hierarchy found under the input (or default) search folder.
  - Write all results and data files associated with the execution and analysis of each run to its analagous folder under `itk_runs`
 
#### Inputs

As with the general LBM Verification Driver, [a configuration file needs to be supplied.](data/itk/example.cfg)
There are extra inputs associated with ITK and geometry manipulation using ITK.
The current supported format is [nrrd](http://teem.sourceforge.net/nrrd/format.html)
But any ITK supported format can easily be added.
The nrrd format provides the ability to 'voxelize' our geometry, and associate a boundary label with each voxel.
We have not yet automated the process from conversion from a surface or volumetric mesh to a lattice, but this conversion can be completed by the user. The location labels are as follows:

  - Inactive : 0
  - Wall : 1-5
  - Inlet : 11-12
  - Outlet : 20

This geometry file coupled with the config file are required for simulation execution.

[A mapping from the nrrd labels to the LBM can specified using our API](drivers/itk/LBMITKDriver.cpp#L79)

To create a nrrd file from imagery, we used [Slicer](https://www.slicer.org/)

#### Outputs

A VTR file is generated as output.
To view the VTR file, use [ParaView.](https://www.paraview.org/)

  - Open ParaView
  - Open the vtr file
  - Apply a threshold to the data on the boundary labels between 0 and 100 and click apply
    - This should now show your geometry
  - You can then select the data you would like to view and continue to process it using the ParaView tools

