The “Internet of Things”, IOT, is a big topic and involves many important concepts about systems, e.g., open vs closed, organized, self-organizing, closed and open-loops. All are worthy topics to understand to put a foundation under IOT. However, I will resist these topics and only discuss Piraeus and the role that Orleans has in the architecture for the sake of brevity.

Piraeus is a multi-channel, multi-protocol, in-memory event broker. It enables edge devices or services to connect and transmit or receive information from other system entities without any coupling and without system entities having direct knowledge of each other. It is a high throughput, low latency, and linearly scalable Operational Technology that simplifies the ability for an open-system to achieve its goal.

Diagram of the physical architecture used in Pegasus-I, i.e., how we got the real-time out Piraeus and Orleans.

The Piraeus Architecture (Operational Technology) that was used for Pegasus-I

A system can be modeled as a directed graph and such Piraeus uses this simple construct to enable communications through its gateway to components within a system or sub-system. Information enters Piraeus through its gateway, then is distributed throughout the graph. The leaves of the graph are connected to either active or passive system components that receive the information.

The components of Piraeus are relatively simple. A gateway exists that can receive information through a variety of channels and protocols. Once this information is received by the Piraeus gateway it is fed into the appropriate graph for that specific information topic. The node that receives this information is a virtual actor, called a grain inside of the Orleans host process. This type of grain is considered a “topic” within Piraeus and topics are graphed to “subscription” grains, which Orleans can communicate. Once the information enters the topic grain is fanned out to all subscription grains associated with the topic. The subscription grains are observable and feed information directly and immediately to the specific channel and protocol that is associated with the subscription. There is no polling in the entire Piraeus architecture…ever.

That is a high-level summary, so let’s dig deeper into what is happening under the covers. Topic grains are always provisioned prior to communications being established. They describe the head node or ingress resource of a sub-graph as part of a system or sub-system. The topic grain’s job is to act as a resource for subscription grains to attach and fulfill the sub-graph enabling transmitter and receiver to communicate unidirectional. This gives us consistency with how a generalized system would behave.

The subscription grains can be either durable or ephemeral. Durable subscription grains are configured with metadata and attached to a topic grain. The metadata is part of the subscription grain-state within Orleans and therefore requires no orthogonal look up or database to manage that would impact performance. Durable subscriptions are used by either active or passive receivers. An active receiver is one that creates a connection to the Piraeus gateway. Once that active connection is established, the identity of the actor is used to create and associate “observables” with the specific subscriptions grain(s) for that identity. When information flows into the subscription grain from the topic grain, the observable for the subscription is already established and associated with the channel and protocol of the receiver. This creates a direct pipeline between transmitter and receiver. A passive receiver is a service that does not connect directly to the Piraeus gateway. When information is passed to these durable subscriptions, the subscription grain will forward the information immediately to the service. Piraeus supports RESTful Web services, Event Hubs, Azure Blob Storage, and Service Bus for these passive subscriptions. This means it is possible to create, e.g., to create Web service, and start receiving information from Piraeus without doing anything else.

Ephemeral subscriptions can be used with self-organizing systems. These are systems that can organically change and system actors can enter and exit arbitrarily. Ephemeral subscriptions only exists for the duration of the active channel. Once the channel is dropped, the subscription grain and its observable are removed and resources are disposed. This enables a self-system to organize without prior knowledge of edge system components. Of course, access control plays a major component on what is allowed, which is not a topic for this post.

Orleans is a radically different type of technology, which is a key enabler for the Pegasus-I mission to near space. It makes possible some of the key concepts around flowing information at low latency and organizing graphs through its concept of virtual in-memory actors, i.e., grains. Observables in Orleans also give Piraeus a simple way to mate specific receivers to their respective subscription grains without need to leverage any store and forward technologies and makes for a remarkably elegant end-2-end communications story that is simple to use. We were able to get telemetry from Pegasus-I inflight to the Web site for a user to view in about 20 milliseconds on average, going from a radio transmission from Pegasus to our field gateways and over MiFi on our phones to Piraeus. Of course, we also proved it also works at 85,000 feet, 2.2% atmosphere, and -60 degrees Fahrenheit.

Notable Technical Details

Orleans is a high throughput, low latency, and linearly scalable technology that enables Piraeus to easily manage a system graph or sub-graph to enable communications between system components.
Subscription grains are either durable or ephemeral in the context of Piraeus.
Subscription grains can be associated with either active or passive receivers.
Piraeus enables systems to be either organized or self-organizing.
Information flow is in-memory by default, but can be persisted if required.
Piraeus uses no databases or storage accounts to perform its operations.
Orleans grain-state is persisted in a customized Orleans Storage Provider that leverages Redis.
Piraeus has a multi-channel and multi-protocol gateway that does not couple channel and protocol between transmitter and receiver.
There is no intrinsic limitation to message size within Piraeus

There is a tremendous amount more about the technical detail, which is far to overwhelming for this overview post. I will try to post more fined-grained detail at a later date for those interested.

Best Wishes,

-Matt Long