# Purpose of this documentation

This documentation covers the lay of the land at the time
of writing. It is a top-down description of asynchronous paraview
w.r.t communication and threading. It starts off with a description of
the current design using RPCs. However, the concepts presented here build
up to a more general form i.e, messaging across threads.
Finally, it captures the asynchronous-networking requirements for an
asynchronous ParaView-based visualization app. For developers that wish
to provide alternative implementations of the `Services` framework,
these requirements are your best friend.


- [Purpose of this documentation](#purpose-of-this-documentation)
- [High level description of start up](#high-level-description-of-start-up)
- [Communication mechanisms](#communication-mechanisms)
  - [Message](#message)
  - [Ask and ye shall receive](#ask-and-ye-shall-receive)
  - [Publish-Subscribe](#publish-subscribe)
- [Communication in context of ParaView application](#communication-in-context-of-paraview-application)
- [Building blocks](#building-blocks)
  - [Module: ParaView::ServicesCore](#module--paraview--servicescore)
  - [Module: ParaView::ServicesThallium](#module--paraview--servicesthallium)
  - [vtkPacket - data packet for communication](#vtkpacket---data-packet-for-communication)
  - [vtkService: a data packet destination](#vtkservice--a-data-packet-destination)
  - [vtkServiceEndpoint: an endpoint to connect to a service](#vtkserviceendpoint--an-endpoint-to-connect-to-a-service)
  - [vtkServicesEngine: application specific session](#vtkservicesengine--application-specific-session)
  - [vtkProvider: add specific capabilities to a service](#vtkprovider--add-specific-capabilities-to-a-service)
- [Coordination of RPCs](#coordination-of-rpcs)
  - [Low level description of start up and communication mechanisms](#low-level-description-of-start-up-and-communication-mechanisms)
  - [Fundamental requirements](#fundamental-requirements)
    - [send_message](#send-message)
    - [send_request](#send-request)
    - [subscribe](#subscribe)
    - [unsubscribe](#unsubscribe)
    - [publish](#publish)
    - [handshake](#handshake)

<small><i><a href='http://ecotrust-canada.github.io/markdown-toc/'>Table of contents generated with markdown-toc</a></i></small>

# High level description of start up

In a typical setup, an application starts up a number of _services_
and sets up message handlers for each _service_. The type of handlers
help establish roles for each _service_ within the application.

The application then creates _endpoints_ that connect to each
_service_. These _endpoints_ support three different communication
mechanisms.

# Communication mechanisms

This project takes a service-based approach in order to support
concurrent execution of VTK pipelines either across threads or processes.

> Ex: A _service_ can crunch numbers with vtk filters, another
> _service_ can render surface/volumetric geometry and yet another
> _service_ can handle IO requests.

Each _service_ runs code sequentially in a separate _thread_ or
_process_. A _process_ can spawn multiple _threads_. In this
architecture, each _service_ executes at its own pace. However,
these _services_ need to exchange information every now and then
so that a data visualization app can do meaningful work!
The information exchange can happen in three mechanisms:

## Message

A _service_ can receive messages from a _service-listener_ or _service-consumer_.
The communication to the _service_ is done through an _endpoint_.

## Ask and ye shall receive

Instead of simply sending a message, Service A sends a request to service B and receives a reply.
Service B may want to handle a _response_ asynchronously without blocking its execution.
Service B can wait some time for a response and handle timeouts gracefully.
This communication is also happens through an _endpoint_.

## Publish-Subscribe

Instead of asking for information, a _service_ or an application can _subscribe_ to a
_channel_. Another _service_ can decide to _publish_ messages onto
that channel. It is useful when a _service_ wishes to receive a _stream_
of data as it becomes available from another _service_. This model of
communication is a form of direct _service_-_service_ communcation.

# Communication in context of ParaView application

This section brings to light messaging mechanisms that are used in the context
of ParaView before diving into the guts of `Services` framework.
It addresses these questions.

> How are proxy properties updated on the remote VTK object?

> How is a VTK pipeline/view update executed?

> How does the data service send geometry to the render service?

> How does an application receive rendered images from the render service?

After a proxy property is modified and pushed through the `PropertyManager` microservice,
control reaches `vtkSMProxy::UpdateVTKObjects`. Here, we pack up all modified
properties along the with the new values into a neat immutable object and ship
it to the destination service. We call this immutable object a `vtkPacket`.
The destination service will need additional knowledge to process such packets.
For example, does the packet contain a state update? Does it contain commands?
Does it update information? Should it delete the remote object?
Therefore, the packet is tagged with a string that describes the type of message.
In the case of updating a property, the tag is `vtk-remote-object-update-state`
and the global id (`gid`) of the proxy is also set to the appropriate proxy
which receives the state update.

`vtkRemoteObjectProvider` and `vtkRemoteObjectProviderViews` help control a
remote VTK object from the client application by providing methods that tag a
`vtkPacket`. They do much more than that, however let's not get into that here.

Tags are very useful when you plan to introduce new functionality in a service.
Existings tags such as `vtk-remote-object-update-information`,
`vtk-remote-object-update-pipeline`, `vtk-remote-object-delete`,
`vtk-remote-object-invoke-command`, `vtk-remote-object-execute-command`,
`vtk-remote-object-gather-information`, `vtk-remote-object-render` aid the data,
render services to process incoming `vtkPacket`(s)

In some cases, a service is interested in the result of the remote _command_.
This is the case with pipeline/view updates. For example, to update a pipeline,
one calls `Proxy.Update()` that results into a message tagged with
`vtk-remote-object-update-pipeline`. Upon arrival, the service will call
`Update()` of the corresponding vtk object.

# Building blocks

This section goes one step deeper. It outlines the low-level communication mechanisms
that make asynchronous ParaView possible.

## Module: ParaView::ServicesCore

`ServicesCore` defines the building blocks for our service-based
approach. It provides an abstraction of a service, endpoint,
provider and packet. If one were to implement asynchronous services with
an alternative rpc or communication library, that new module must declare
object factory overrides for the classes in `ServicesCore` module.

## Module: ParaView::ServicesThallium

`ServicesThallium` is a concrete implementation of the `ServicesCore` module.
[Thallium](https://mochi.readthedocs.io/en/latest/thallium.html)
is used to provide asynchronous RPC functionality. Any other rpc
library can have it's own factory overrides in a separate
`ServicesRPCLibraryName` module.

## vtkPacket - data packet for communication

`vtkPacket` represents a data unit. The services exchange information by moving around `vtkPacket`(s).
It is essentially a json object with `std::string` keys and type agnostic values.

`vtkPacket` is immutable by design; once constructed, it cannot be modified.
This is an important design consideration that lets the implementation minimize
thread-safety concern while avoiding data copies. As the packet floats across
threads and execution contexts, we're assured that its contents are never
changed.
By implication, if a pointer, for example a `vtkDataObject*`, is stored in a
packet, the item pointed to should also be considered immutable throughout the
lifetime of the packet and any of its copies. Conceptually, once a packet is
sent on the "wire", you cannot change it.

## vtkService: a data packet destination

`vtkService` is the de facto destination for all data packets. It can be thought
of a service in the conventional sense. Besides API to start/stop the service,
`vtkService` also provides API to setup handler for messages sent to the
service. Additionally, it provides API to push data to a named channel.

## vtkServiceEndpoint: an endpoint to connect to a service

`vtkServiceEndpoint` represents an endpoint to communicate with a
`vtkService`. It provides API to send data packets to a connected service and
receive responses from it.

## vtkServicesEngine: application specific session

`vtkServicesEngine` keeps track of services and service-endpoints.

## vtkProvider: add specific capabilities to a service

`vtkProvider` is simply an abstract base class intended to be subclassed to add
specific capabilities to a service. The message tag is handled here and the information
in the packet translates to an action. In other words, it is a dispatcher that
analyzes the message and invokes the appropriate command.

# Coordination of RPCs

On every process, there is an instance of (`vtkServicesEngine`) that manages services.
This _engine_ always has one thread that processes rpc requests.
Each _service_, in turn has it's own run loop (on the service thread) that processes
incoming requests.
RPC requests are received in the engine' dedicated RPC thread,
and enqueued on the destination _service_ thread's
run loop. The `rxcpp` library establishes certain rules that govern
emission and capture of events across multiple threads.

> The _RPC_ thread schedules a function call. Service thread
> waits to be notified by _RPC_ thread. A _service_ run loop
> drives this action by popping the event queue when necessary.

## Low level description of start up and communication mechanisms

This description of start up sequence answers the following questions.

> What does a services engine do?

> How is a service created?

> How does one obtain an endpoint to a remote service?

> How does a client connect to a remote service through an endpoint?

> What rpc facilitates endpoint connection?
> Where is that rpc defined?

> How does a client send a message to another service through an endpoint?
> What rpc facilitates this?
> Where is it defined?

> How does a client send a message to another service through an endpoint
> and request a response?
> What rpc facilitates this?
> Where is it defined?

> How does pub-sub work between two services? What rpc(s) facilitate this? Where are they defined?

The application creates a services engine instance and hooks it with the main thread's scheduler.
This scheduler can be used to observe/subscribe rxcpp events on the application's main thread.
New services and endpoints can be created with the engine.
The engine also initializes a service after creation, which simply ends up naming the service
and launching a new thread bound to the service with an rxcpp run-loop.

In a client-server style application, some services live only on the client
or only the server. Communication among these services can take two possible routes,
depending on the location of the two services in discussion.
When both services belong to the same engine i.e, either the engine on client or engine on server,
they can directly communicate with `vtkService` API. However, a more common usage
involves communicating between client and server. In this case, all requests
and responses go through `vtkServiceEndpoint`.

Recall that endpoints can communicate with a remote service. Before an application can use an endpoint,
it needs to be connected to the remote service. The connection procedure has a simple goal

- obtain a handle to the remote service from the engine on the other side.

> In the thallium implementation, the `handshake` rpc defined in the engine
> looks up a unique ID for the service and returns it.

In this illustration, server and client do not share the same address space,
both possibly run on different machines or even different OS or architecture.

```cpp
+---------------------------------+       +-----------------------------------------+
|   Server                        |       |   Client                                |
+---------------------------------+       +-----------------------------------------+
| Engine engine                   |       |  Engine engine                          |
| engine->CreateService("s1")     |       |  engine->CreateService("sa")            |
| engine->CreateService("s2")     |       |  engine->CreateService("sb")            |
| engine->CreateService("s3")     |       |                                         |
|//engine keeps record of services|       |//engine keeps record of services        |
| ids: {"s1": 1, "s2": 2, "s3": 3}|       | {"sa": 1, "sb": 2}                      |
|                                 |       |                                         |
|     /* Illustration of endpoint creation, connection and the handshake RPC*/      |
|rpc:handshake(string serviceName)|       |                                         |
|{                                |       |s1Endpoint = engine->CreateEndpoint("s1")|
| // look up handle to a service  |       | // calls rpc:handshake on server        |
| handle = ids["s1"]              |       |s1Endpoint->Connect().Wait()             |
| respond with handle             |       |                                         |
|}                                |       |                                         |
|                                 |       |                                         |
+---------------------------------+       +-----------------------------------------+
```

What you may notice above is that an endpoint simply calls an RPC that is defined somewhere
known to the engine implementation.
This mechanism is pervasive in the services module.

- Sending a message boils down to invoking an RPC (defined in the destination service) with the message as an argument.
- Requesting a reply is a slight modification -the RPC now returns a value.
- Publishing data from a service boils down to invoking an RPC defined in a connected service endpoint. This RPC accepts two arguments: channel and data. All it does is push the data into a stream corresponding to the channel.
- Subscribing to a channel on a service ends up creating an rxcpp pipeline that performs the above step every time data is pushed into the channel observable.

## Fundamental requirements

In the initial stages of this project, the tags mentioned in
[Communication in context of ParaView application](#communication-in-context-of-paraview-application)
were actual RPCs that were invoked by application and the various services. Over time,
the architecture slimmed down to use a tag-based system. RPC calls concerning VTK objects
has been moved to `vtkProvider` and the `vtkServicesCore` layer holds only infrastructure
RPCs - 6 in total i.e, `send_message`, `send_request`, `subscribe`, `unsubscribe`, `publish`
for the core functionality and `handhshake` for the initial connection.

### send_message

An endpoint must be able to send a message to another service. However, the endpoint send method should not block. The engine must be able to queue requests to send messages
on the thread that invokes a network call.

### send_request

When a service responds to a message, the engine should be capable of receiving such replies.

### subscribe

The endpoint or another service should be able to receive messages with a named channel.

### unsubscribe

The endpoint or another service should be able to stop receiving messages on a named channel.

### publish

A service should be able to send a message with a channel key to subscribed endpoints.

### handshake

The engine should verify a connection secret, retrieve and return an internal handle
to the service. This handle shall help an endpoint communicate with the remote service.

Besides these fundamental requirements, there are technical requirements of having
bidirectional communication over a single socket.
