More

Communication Protocols

The communication process between  applications, through a computer network,  can follow two well-defined  paths, as showed in the next figure.

In the first way, using APIs (sockest, for example), the application requests a communication service that is provided by a protocol stack   (TCP/IP, for example) implemented as part of the operating system. In this path, each layer of the protocol stack processes the message, that is transfered from buffer to buffer until it be inserted in the interconnection physical medium. In addition, we have some context switching between processes and a lot of function calls. All of this produces high overheads that cause great latencies and loss of bandwidth.  For networks based in slow phisical mediuns, this overhead has little influence in the overall performance.

On the other side, the second path followed by messages is optimized to reduce the disadvantages of the first. The communication process begin like the first, but the communications services are provided by a Basic Communication Layer (BCL), implemented as library functions, that makes a direct connection with the NIC or with its device driver. Thus: a) we don't have system calls in the message's path; b) the messages avoid to cross a lot of software layers, and c) the numbers of message copies is decreased. As a result, we get a communication with low latency and   better bandwidth utilization. This approach is very important, mainly when the physical interconnection network  has low latency and high bandwidth, such as System Area Networks (SAN) used in computer clusters.

Back

Light-weigth Protocols

Ligth-weigth protocols, also called user level protocols, feature low overhead. Theses protocols implement basic functions to support communications and, usually, are designed for fast interconnection systems with low error rates.

The basic principle is to avoid OS intervention when messages are sent or received, except when strictly necessary to open or close connections or to control some parameters.

Among the most important researches, we can highlight:

• Active Messages (AM): the fundamental ideia is to place, in a message header, a pointer to a function (message handler) that is executed in user space as soon as the message is received. Active Message was conceived by researchers of University of California, under leadership of Thosten von Eicken.

• Fast Messages (FM): This communication layer employes similar concepts to AM (message handler). However, FM has better support to high level communication level (sockets, MPI, PVM). The FM project was conduced by Scott Pakin at University of Illinois.

• Basic Interface for Parallelism (BIP): The BIP protocol presents only the strictly necessary operations for a basic communication layer. In this way, the BIP protocol has a reliabity level determined by the reliability level of the hardware, and it doesn't provide any mechanism to recover from errors or losses. The researchs in this area are headed by Loic Prylli at Lyon University.

• U-Net Project: The essence of this project is to remove the OS from the path of send/receive messages by employing the virtual interface concept. So, using virtual interface, each application has its own instance of the NIC that can be accessed in a direct and safe way. This project was developed in the University of Cornell and it was headed by Thorsten von Eicken and Matt Welsh. This communication architecture had strong influences over the VIA architecture.

Back

Virtual Interface Architecture (VIA)

The VIA protocol has many similarities with U-Net architecture. Actually, VIA is un open specification of a communication layer, delivered as join work of several computer industries headed by Intel, Compaq and Microsoft. The following picture illustrates the communication model of VIA.

Before a messages is sent, it is necessary to create, through calls to OS kernel, VIA instances also  named VI. Each VI has two queues of descriptors, one of send (it describes an region of memory alocated for data to send) and another of receive (it describes a memory region to receive data). A descriptor is a data structure defined is application space, and it holds information about data (address, size). Completed descriptors can be inserted in a Completion Queue (CQ) or  they can directly be removed from their particular work queue.

The application can straight ask services from a library of VIA functions (user level library), or by means of an software communication interface (sockets, MPI, PVM). This APIs make function calls to the VIA library. This approach improves portability of existing applications as well as the development of new applications using this well-knowon APIs.

Date Transfer Models:

The VIA v1.0 specification states two models for data tranfers:

a) Send/Receive: In this paradigm,  each endpoint (instance VI) involved in the communication process must post a descriptor in its work queue describing a data memory region. The flow control must be done by the concerned applications.

b) Remote Direct Memory Access (RDMA):  In this case, the iniciator of communication must specify both the address of the source buffer and the destination buffer of the data transfer. We have two types: RDMA-Write and RDMA-Read.
   The source of the RDMA-Write operation can be specified as a gather list of buffers, while the destination must be a single and contiguous region of virtual memory. On the other side, in the RDMA-Read, the source, at the remote end,  must be specified as single and contiguous area of virtual memory, while the destination can be specified as a scatter list of buffers.

Reliability Levels:

There are three levels: Unreliable Delivery, Reliable Delivery and Reliable Reception. The following table shows the main characteristics.

Table: Summary of VIA Reliability Guarantees (Conform to VIA Specification v1.0).

Back