README.md 5.89 KB
Newer Older
Berk Geveci's avatar
Berk Geveci committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# Introduction

Extreme scale scientific simulations are leading a charge to exascale
computation, and data analytics runs the risk of being a bottleneck to
scientific discovery. Due to power and I/O constraints, we expect _in situ_
visualization and analysis will be a critical component of these workflows.
Options for extreme scale data analysis are often presented as a stark contrast:
write large files to disk for interactive, exploratory analysis, or perform in
situ analysis to save detailed data about phenomena that a scientists knows
about in advance. We present a novel framework for a third option – a highly
interactive, image-based approach that promotes exploration of simulation
results, and is easily accessed through extensions to widely used open source
tools. This _in situ_ approach supports interactive exploration of a wide range of
results, while still significantly reducing data movement and storage.

More information about the overall design of Cinema is available in the paper,
An Image-based Approach to Extreme Scale In Situ Visualization and Analysis,
which is available at the following link:
[https://datascience.lanl.gov/data/papers/SC14.pdf](https://datascience.lanl.gov/data/papers/SC14.pdf).

This repository contains a set of Python modules that make it easy to read
and write Cinema databases of various specs and formats. Our goal is to grow
this API as new functionality is added to Cinema by the community.

# Requirements

* Python 2.x
* numpy
* PIL (not required for output if VTK is available)
* VTK (optional)
* ParaView (optional)
32
33
34
* scipy (only required for unit tests)
* OpenEXR (optional)

35
36
37
38
39
40
# Contributing

See [CONTRIBUTING.md][CONTRIBUTING.md] for instructions to contribute.

[CONTRIBUTING.md]: CONTRIBUTING.md

41
42
43
# Installation of OpenEXR

OpenEXR 1.2.0 is supported as a database image format (currently only for depth images). This version of OpenEXR is readily available as a python egg and as a regular python module.
44
The module can be installed as a [python egg][OpenEXREgg] or as a [regular module][OpenEXRModule].
45

46
47
[OpenEXRModule]: https://pypi.python.org/pypi/OpenEX://pypi.python.org/pypi/OpenEXR
[OpenEXREgg]: http://excamera.com/sphinx/articles-openexr.html#openexrpython
Berk Geveci's avatar
Berk Geveci committed
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

# Basic Usage

If you have numpy, PIL and VTK installed, you can use one of the tests
to generate a simple Cinema database as follows.

```python
>>> from cinema_python import tests
>>> tests.test_vtk_clip("./info.json")
```
This should generate a number of png files and one json files that looks
similar to this:

```json
{"associations": {}, "arguments": {"theta": {"default": -180, "values": [-180, -140, -100, -60, -20, 20, 60, 100, 140, 180], "type": "range", "label": "theta"}, "phi": {"default": 0, "values": [0, 40, 80, 120, 160], "type": "range", "label": "phi"}, "offset": {"default": 0, "values": [0, 0.2, 0.4, 0.6, 0.8, 1.0], "type": "range", "label": "offset"}}, "name_pattern": "{phi}_{theta}_{offset}_slice.png", "metadata": {"type": "parametric-image-stack"}}
```

Berk Geveci's avatar
Berk Geveci committed
65
For details about the Cinema spec used, see the [Cinema specs repo](https://gitlab.kitware.com/cinema/specs).
Berk Geveci's avatar
Berk Geveci committed
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120

You can view the generated database using the [Python Qt viewer](https://gitlab.kitware.com/cinema/qt-viewer) or the
[basic Web viewer](https://gitlab.kitware.com/cinema/basic-web-viewer).

Here is a simplified version of the code from `test_vtk_clip`.

```python
import explorers
import vtk_explorers
import vtk

# Setup a VTK pipeline
s = vtk.vtkSphereSource()

plane = vtk.vtkPlane()
plane.SetOrigin(0, 0, 0)
plane.SetNormal(-1, -1, 0)

clip = vtk.vtkClipPolyData()
clip.SetInputConnection(s.GetOutputPort())
clip.SetClipFunction(plane)
clip.GenerateClipScalarsOn()
clip.GenerateClippedOutputOn()
clip.SetValue(0)

m = vtk.vtkPolyDataMapper()
m.SetInputConnection(clip.GetOutputPort())

rw = vtk.vtkRenderWindow()
r = vtk.vtkRenderer()
rw.AddRenderer(r)

a = vtk.vtkActor()
a.SetMapper(m)
r.AddActor(a)

# make or open a cinema data store to put results in
cs = FileStore(fname)
# The png files created will be named using this pattern
cs.filename_pattern = "{phi}_{theta}_{offset}_slice.png"
# We add 3 parameters: 2 camera angles & the slice offset
cs.add_parameter("phi", make_parameter('phi', range(0, 200, 40)))
cs.add_parameter("theta", make_parameter('theta', range(-180,200,40)))
cs.add_parameter("offset", make_parameter('offset', [0,.2,.4,.6,.8,1.0]))

# This explorer will change the camera parameters (phi and theta) during generation
cam = vtk_explorers.Camera([0,0,0], [0,1,0], 3.0, r.GetActiveCamera()) #phi,theta implied
# Changes the slice offset
g = vtk_explorers.Clip('offset', clip)
# Combines all explorers to generate the images
e = vtk_explorers.ImageExplorer(cs, ['offset','phi', 'theta'], [cam, g], rw)

# run through all parameter combinations and put data into the store
e.explore()
```
Berk Geveci's avatar
Berk Geveci committed
121

David DeMarle's avatar
David DeMarle committed
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
The cinema module is also the recommended path for working with cinema stores that have been generated. The following code demonstrates the API for opening a store, searching for elements within it, and doing some trivial analysis of them.

```python
from cinema_python import cinema_store
import PIL.Image
import sys

def demonstrate_analyze(fname):
    """
    this demonstrates traversing an existing cinema store and doing some analysi
    (in this case just printing the contents) on each item"""

    cs = cinema_store.FileStore(fname)
    cs.load()

    print "PARAMETERS ARE"
    for parameter in cs.parameter_list:
        print parameter
        print cs.get_parameter(parameter)['values']

    print "ONE PARAMETER'S FIRST VALUE IS"
    param = cs.parameter_list.keys()[0]
    val = cs.get_parameter(param)['values'][0]
    print val

    print "HISTOGRAMS OF MATCHING RECORDS FOR", param, "=", val, "ARE"
    for doc in cs.find({param:val}):
        print doc.descriptor
        image = PIL.Image.fromarray(doc.data)
        print image.histogram()

if len(sys.argv) != 2:
  print "Usage: python demo.py /path/to/info.json"
  exit(0)

demonstrate_analyze(sys.argv[1])
```

Berk Geveci's avatar
Berk Geveci committed
160