Commit c4dc0bb3 authored by Mike Rye's avatar Mike Rye

Initialized.

parents
__pycache__
This diff is collapsed.
# About
The following instructions will allow the user to simply move their head to peer into the active [ParaView](https://www.paraview.org/) scene at different angles. This is achieved by running a socket server in a separate terminal to fetch webcam images and a QTimer in ParaView's Python shell to adjust the view in real-time. The adjustment algorithm uses [OpenCV](https://opencv.org/)'s Python bindings and runs in ParaView's Python shell.
# Dependencies
* [ParaView](https://www.paraview.org/)
* [OpenCV](https://opencv.org/) with Python bindings.
* [PyQt](https://riverbankcomputing.com/software/pyqt/) compatible with the version of Qt used by ParaView.
You will also need a webcam.
# Quick Start (POSIX)
Just run the "paraview_wrapper.sh" script and follow the instructions on the terminal and in the Python shell.
# Detailed Instructions
The webcam must have a view of the user's face as they look at the screen. The angles should be minimized so that the user's face is as close to the camera's optical axis as possible when in front of the screen, e.g. by placing the camera at the top of the screen.
## Server
The server should be launched in a different terminal. A resolution of 640 x 480 pixels is both sufficient for face-tracking and fast enough to remain responsive to the user's movement.
```sh
./webcam_server.py -s 640x480
```
Use the "-d" option to select a different device if necessary, e.g. By default, the server will connect to the first webcam (e.g. /dev/video0 on Linux).
```sh
./webcam_server.py -s 640x480 -d /dev/video1
```
## ParaView
To import the necessary modules in the Python shell, the "src" directory must be added to the `PYTHONPATH` environment variable before launching ParaView, which must be done within the variables scope. For example, in a POSIX terminal:
~~~~sh
export PYTHONPATH=/full/path/to/src:$PYTHONPATH
paraview
~~~~
Once ParaView is running, load or create a scene with data that you want to view then open the Python shell: View- > Python Shell. Run the following commands in the shell to create a `CameraAdjuster` object to start and stop the face-tracking service:
```python
import adjust_camera
cam = GetActiveCamera()
ca = adjust_camera.CameraAdjuster(cam, Render)
```
The first line imports the module, the second gets the current scene's camera and the third creates the `CameraAdjust` object `ca` with the scene's camera and `Render` function, which is needs to update the scene. Note that **if you change scenes**, it will be necessary to run the second and third steps again to get the new active camera.
To start the service, simply run `ca.start()` and the view should begin following the user's movements. Run `ca.stop()` to stop the service.
While the service is running, the user can manually adjust the camera with the mouse as usual to configure the view. This is also a "hands-free" mode that can be activated by stopping the service, executing `ca.hands_free = True` and then starting the service again. When this mode is set, the view will begin to continuously rotate in the direction of the user's face if it moves far enough to any side. The camera will also zoom in and out as the user's face moves closer to or farther from the camera, respectively. To de-activate this mode, stop the service as run `ca.hands_free = False` before starting it again.
If you want to tweak the code without restarting ParaView, use `reload` from `importlib` to load your changes:
```python
ca.stop()
from importlib import reload
reload(adjust_camera)
ca = adjust_camera.CameraAdjuster(cam, Render)
ca.start()
```
The commands to load the module, create the `CameraAdjuster` object and import `reload` are provided in `pv_console_cmds.py`. They can be loaded in the Python shell by running `exec(open('pv_console_cmds.py').read())` if ParaView was launched in the same directory.
## Configuration File
For the best performance, the parameters in the configuration file (`dat/camera_adjuster.conf`) will likely need to be changed. Please see the comments in that file for details.
# Troubleshooting
* If tracking does not work, make sure that the user's face is illuminated enough to be clearly visible to the camera and that the camera is in front of the user. The user's face and the camera should also share the same vertical orientation (i.e. up the for face should be up for the camera).
* If the view begins to wobble, try restarting and maybe recreating the camera adjuster.
[camera]
# Camera angle [degrees].
view angle = 15
[display]
# The height of the viewport in which the data is shown [centimeters].
height = 32
[face]
# Average face width [centimeters].
width = 11
# Average face distance from display [centimeters].
distance = 65
# The threshold distance from the average distance along the camera axis beyond
# which the view will begin to zoom when in "hands-free" mode [centimeters].
normal distance threshold = 10
# The threshold distance from the camera axis beyond which the view will begin
# to rotate when in "hands-free" mode [centimeters].
perpendicular distance threshold = 10
[other]
# The number of position samples to average.
samples = 7
# The path to the facial Haar cascade file.
# https://github.com/opencv/opencv/blob/master/data/haarcascades/haarcascade_frontalface_default.xml
haar cascade = dat/haarcascade_frontalface_default.xml
# The face-display distance is calculated using a simple formula with 2
# parameters:
#
# distance = (a / face_width) - b
#
# These parameters depend on the camera. The following values were determined
# for a Logitech HD 1080p by fitting 3 points. Other setups may need to adjust
# these.
parameter a = 1.04e4
parameter b = 2.74
This diff is collapsed.
#!/bin/bash
set -eu
if ! command -v paraview
then
echo 'Error: failed to find ParaView. Make sure that the directory with the "paraview" executable is in your PATH environment variable.'
fi
# Get the absolute path to the "src" directory in the same directory as this
# file.
SELF=$(readlink -f "$0")
ROOT=${SELF%/*}
SRCDIR=$ROOT/src
# # Set the PYTHONSTARTUP environment variable so the console commands are
# # executed automatically.
export PYTHONSTARTUP=$ROOT/pv_console_cmds.py
# Set or prepend the Python path depending on whether it's empty.
if [[ -z ${PYTHONPATH:+x} ]]
then
export PYTHONPATH=$SRCDIR
else
export PYTHONPATH=$SRCDIR:$PYTHONPATH
fi
# Launch the camera in the background.
"$SRCDIR/webcam_server.py" -s 640x480 &
# Set a trap to ensure that the webcam server is killed when ParaView is shut
# down.
trap "kill $!" EXIT
# PYTHONSTARTUP doesn't work because the Python shell in ParaView is an emulator
# and not a true shell. Display a message instead.
cat << MSG
Open the Python shell (View -> Python Shell) and run the following command:
exec(open('$PYTHONSTARTUP').read())
MSG
# Launch ParaView with an passed arguments.
paraview "$@"
# Load these commands in the ParaView Python shell with
#
# exec(open('path/to/this/file').read())
#
from importlib import reload
import adjust_camera
cam = GetActiveCamera()
ca = adjust_camera.CameraAdjuster(cam, Render)
print('''USAGE
Start the face-tracking service:
ca.start()
Stop the face-tracking service:
ca.stop()
Reload changes in the adjust_camera module:
ca.stop()
reload(adjust_camera)
ca = adjust_camera.CameraAdjuster(cam, Render)
ca.start()
Update camera after changing views:
ca.stop()
cam = GetActiveCamera()
ca = adjust_camera.CameraAdjuster(cam, Render)
ca.start()
Activate "hands-free" mode (see the README for details):
ca.stop()
ca.hands_free = True
ca.start()
''')
#!/usr/bin/env python3
import cv2
import numpy as np
import vtk
import vtk.vtkFiltersGeneralPython as filters
from webcam_server import WebcamServer
class ViewTracker:
def __init__(
self,
webcam_server,
face_cascade,
distance_func,
face_width,
average = 3
):
self.webcam_server = webcam_server
self.face_cascade = face_cascade
self.distance_func = distance_func
self.face_width = face_width
self.average = average
self.positions = np.zeros((average, 3))
self.positions[:,2] = 1
self.index = 0
self.last_face_pos = np.array((0, 0))
def add_position(self, w_img, h_img, x, y, w_face, h_face):
'''
Given an image width and height and the x, y, width and height of a detected
face's bounding box, calculate the position vector and add it to the
internal list of positions.
'''
d = self.distance_func(w_face)
cm_per_pixel = self.face_width / w_face
# TODO: check why h_face is needed to center y_face
# but w_face is not needed for x_face
x_face = x - (w_img / 2)
y_face = y + h_face - (h_img / 2)
r = np.array((x_face * cm_per_pixel, y_face * cm_per_pixel, d))
self.positions[self.index,:] = r
self.index = (self.index + 1) % self.average
def detect_faces(self, img):
if img is not None and img.size > 0:
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
faces = self.face_cascade.detectMultiScale(
gray,
scaleFactor=1.1,
minNeighbors=5,
minSize=(30, 30),
flags = cv2.CASCADE_SCALE_IMAGE
)
if not isinstance(faces, tuple):
return faces
return None
def get_image_with_rectangles(self, img=None):
if img is None:
img = self.webcam_server.retrieve().copy()
faces = self.detect_faces(img)
if faces is None:
return img
for (x,y,w,h) in faces:
cv2.rectangle(img, (x,y), (x+w, y+h), (0, 0xff, 0), 2)
return img
def get_current_position(self, unit=True, img=None):
if img is None:
img = self.webcam_server.retrieve()
faces = self.detect_faces(img)
if faces is not None:
main_face = None
shortest_dist = 0.0
current_face_pos = self.last_face_pos
# Each face is (x, y, w, h).
for face in faces:
face_pos = np.array(face[:2])
face_dist = np.linalg.norm(face_pos - self.last_face_pos)
if main_face is None or face_dist < shortest_dist:
main_face = face
shortest_dist = face_dist
current_face_pos = face_pos
self.last_face_pos = current_face_pos
if main_face is not None:
self.add_position(*img.shape[:2], *main_face)
# Return the averaged position for smooth transitions.
r = np.average(self.positions, axis=0)
if unit:
r /= np.linalg.norm(r)
return r
def get_image_and_position(self, img=None, unit=False):
img = self.webcam_server.retrieve()
img2 = self.get_image_with_rectangles(img=img)
pos = self.get_current_position(img=img, unit=unit)
return img2, pos
def get_current_rotation(self):
r = self.get_current_position()
r0 = np.array((0, 0, 1))
if np.any(r != r0):
return np.cross(r0, r)
else:
return None
def get_vtk_transform(self):
rot_axis = self.get_current_rotation()
if rot_axis is None:
return None
rot_angle = np.arcsin(np.linalg.norm(rot_axis)) * 180 / np.pi
transform = vtk.vtkTransform()
transform.RotateWXYZ(rot_angle, rot_axis)
return transform
def apply_vtk_transform(self, obj):
transform = self.get_vtk_transform()
if transform is None:
return None
transformFilter = filters.vtkTransformFilter()
transformFilter.SetInputData(obj)
transformFilter.SetTransform(transform)
transformFilter.Update()
return transformFilter.GetOutput()
#!/usr/bin/env python3
import configparser
import os
import sys
import time
import cv2
import numpy as np
from PyQt5 import QtCore
#
import ViewTracker #import ViewTracker
import webcam_server #import WebcamServer
#-------------------------------------------------------------------------------
# ConfigError
#-------------------------------------------------------------------------------
class ConfigError(Exception):
def __init__(self, msg):
self.msg = msg
def __str__(self):
return 'ConfigError: {}'.format(self.msg)
#-------------------------------------------------------------------------------
# Config
#-------------------------------------------------------------------------------
class Config():
def __init__(self, path):
# Config file path.
self.path = path
# Camera view angle.
self.view_angle = 15
# Approximate (full F11) view window height (cm).
self.window_height = 32
# Approximate face width (cm).
self.face_width = 11
# Average face-screen distance for angle calculation (cm).
self.face_dist = 65
# Distance from FACE_DIST beyond which the position will begin moving relative
self.norm_dist_threshold = 10
# Distance from center of camera beyond which the display will continue to turn.
self.perp_dist_threshold = 10
# Distance function parameters.
self.distance_params = (1.04e4, 2.74)
# Number of positions to average.
self.pos_avg = 7
self.update_derived()
self.load()
def update_derived(self):
self.half_window_height = self.window_height / 2
self.angle_avg = self.pos_avg * 10
def load(self, path=None):
if path is not None:
self.path = path
if self.path is not None:
parser = configparser.ConfigParser()
try:
parser.read(self.path)
self.view_angle = float(parser['camera']['view angle'])
self.window_height = float(parser['display']['height'])
self.face_width = float(parser['face']['width'])
self.face_dist = float(parser['face']['distance'])
self.norm_dist_threshold = float(parser['face']['normal distance threshold'])
self.perp_dist_threshold = float(parser['face']['perpendicular distance threshold'])
self.pos_avg = int(parser['other']['samples'])
self.haar_cascade_path = parser['other']['haar cascade']
self.distance_params = (
float(parser['other']['parameter a']),
float(parser['other']['parameter b'])
)
self.update_derived()
except (IOError, ) as e:
raise ConfigError(str(e))
except (AttributeError, KeyError) as e:
raise ConfigError('missing entry for {}'.format(e))
except (ValueError, ) as e:
raise ConfigError('incorrect value in config file: {}'.format(e))
def distance_func(self, face_width):
a, b = self.distance_params
return (a / face_width) - b
#-------------------------------------------------------------------------------
# Generic functions.
#-------------------------------------------------------------------------------
def get_rotation_matrix_from_view_up_and_rel_pos(y, z):
z /= np.linalg.norm(z)
# View up is not necessarily perpendicular to the relative position.
y -= np.dot(y, z) * z
y /= np.linalg.norm(y)
x = np.cross(y, z)
return np.vstack((x,y,z)).transpose()
def get_rotation_matrix_from_axis_and_angle(axis, angle):
# http://www.euclideanspace.com/maths/geometry/rotations/conversions/angleToMatrix/
c = np.cos(angle)
s = np.sin(angle)
t = 1. - c
x = axis[0]
y = axis[1]
z = axis[2]
return np.array((
(t*x*x + c, t*x*y - z*s, t*x*z + y*s),
(t*x*y + z*s, t*y*y + c, t*y*z - x*s),
(t*x*z-y*s, t*y*z + x*s, t*z*z + c)
))
#-------------------------------------------------------------------------------
# Camera Adjuster
#-------------------------------------------------------------------------------
class CameraAdjuster():
def __init__(self, camera, update, hands_free=False, config=None):
self.camera = camera
self.update = update
self.timer = QtCore.QTimer()
self.hands_free = hands_free
self.ws = webcam_server.WebcamServer()
# Hard-coded relative default config path for simplicity. If this ever
# becomes an installable project, this will be changed to comply with XDG
# standards.
if config is None:
this_file = os.path.abspath(__file__)
this_dir = os.path.dirname(this_file)
root = os.path.dirname(this_dir)
config = os.path.join(root, 'dat', 'camera_adjuster.conf')
self.config = Config(config)
self.fc = cv2.CascadeClassifier(self.config.haar_cascade_path)
self.vt = ViewTracker.ViewTracker(self.ws, self.fc, self.config.distance_func, self.config.face_width, average=self.config.pos_avg)
self.angle_n = self.config.angle_avg
self.angles = np.ones((self.angle_n,)) * camera.GetViewAngle()
self.angle_i = 0
self.init_pos = np.array(camera.GetPosition())
self.init_view_up = np.array(camera.GetViewUp())
self.last_pos = None
self.last_change = time.time()
camera.SetViewAngle(self.config.view_angle)
def __call__(self):
vt = self.vt
camera = self.camera
update = self.update
# Get current focal point and view up.
focal_point = np.array(camera.GetFocalPoint())
# view_up = np.array(camera.GetViewUp())
view_up = self.init_view_up
# Update initial position if the user has changed the view.
if self.last_pos is not None:
actual_pos = np.array(camera.GetPosition())
shift = actual_pos - self.last_pos
self.init_pos += shift
self.init_view_up = camera.GetViewUp()
# The relative position from the camera to the focal point.
rel_pos = self.init_pos - focal_point
unit_rel_pos = rel_pos / np.linalg.norm(rel_pos)
# Calculate the rotation matrix to convert the camera's frame to the view's frame.
view_rot_matrix = get_rotation_matrix_from_view_up_and_rel_pos(view_up, unit_rel_pos)
# Get the current position of the main face. The non-unit vector will be used below.
try:
r_cm = vt.get_current_position(unit=False)
except ConnectionRefusedError as e:
print(e)
return
r = r_cm / np.linalg.norm(r_cm)
# The camera's unit normal vector.
r0 = np.array((0, 0, 1))
# Calculate the rotation matrix based on the face's position relative to the normal.
if np.any(r != r0):
dr = r0 - r
rel_rot_axis = np.cross(r, r0)
rel_rot_axis_norm = np.linalg.norm(rel_rot_axis)
rel_rot_axis /= rel_rot_axis_norm
rel_rot_angle = np.arcsin(rel_rot_axis_norm)
rel_rot_axis = np.dot(view_rot_matrix, rel_rot_axis)
rot_matrix = get_rotation_matrix_from_axis_and_angle(rel_rot_axis, rel_rot_angle)
else:
rot_matrix = np.identity(3)
# Update position.
new_rel_pos = np.dot(rot_matrix, rel_pos)
new_pos = focal_point + new_rel_pos
camera.SetPosition(new_pos)
unit_new_rel_pos = new_rel_pos / np.linalg.norm(new_rel_pos)
new_view_up = view_up - np.dot(view_up, unit_new_rel_pos) * unit_new_rel_pos
new_view_up /= np.linalg.norm(new_view_up)
camera.SetViewUp(new_view_up)
self.last_pos = new_pos
if self.hands_free:
t = time.time()
dt = t - self.last_change
# Continue turning if face is outside of radius.
planar_r_cm = r_cm.copy()
face_screen_dist = r_cm[2]
planar_r_cm[2] = 0.
planar_dist = np.linalg.norm(planar_r_cm)
if planar_dist > self.config.perp_dist_threshold:
dangle = (np.pi / 7.5) * dt
overshoot_rot_matrix = get_rotation_matrix_from_axis_and_angle(rel_rot_axis, dangle)
self.init_pos = np.dot(overshoot_rot_matrix, rel_pos) + focal_point
ddist = face_screen_dist - self.config.face_dist
if np.abs(ddist) > self.config.norm_dist_threshold:
direction = self.init_pos - focal_point
direction /= np.linalg.norm(rel_pos)
self.init_pos += np.sign(ddist) * direction * dt
self.last_change = t
update()
def adjust(self):
self()
def start(self):
self.timer.timeout.connect(self.adjust)
self.timer.start(0.2)
def stop(self):
self.timer.stop()
def reset(self):
self.init_pos = np.array(self.camera.GetPosition())
#-------------------------------------------------------------------------------
# Simple Testing
#-------------------------------------------------------------------------------
def dummy_update():
print('updated')
class DummyCamera():
def __init__(self):
self.position = (0.,0.,7.)
self.view_up = (0., 1., 0.)
self.angle = 30.0
def GetFocalPoint(self):
print('getting focal point')
return (0,0,0)
def GetPosition(self):
print('getting position')
return self.position
def SetPosition(self, position):
print('setting position:', position)
self.position = position
def GetViewUp(self):
print('getting view up')
return self.view_up
def SetViewUp(self, view_up):
print('setting view up')
self.view_up = view_up
def GetViewAngle(self):
print('getting view angle')
return self.angle
def SetViewAngle(self, angle):
print('setting angle:', angle)
self.view_angle = angle
def test():
ca = CameraAdjuster(DummyCamera(), dummy_update)
while True:
ca()
#-------------------------------------------------------------------------------
if __name__ == '__main__':
try:
test()
except KeyboardInterrupt:
pass
except ConfigError as e:
sys.exit(str(e))
#!/usr/bin/env python3
import argparse
import socket
import sys
import threading
import time
import numpy as np
import cv2