2.8. Avoiding Barriers

Barrier synchronization is appealing because it provides a clear definition of the status of each PE. Although the hardware barrier mechanism on CRAY T3E systems is fast, the waiting time may be long. A barrier requires that all PEs involved arrive at the barrier before any can proceed, so the true speed of a barrier is the speed of the slowest PE. If your application is not well balanced, waiting can slow it down dramatically.

PVM provides a simple form of synchronization. When a PE uses a blocking receive (glossary, ) to receive a message from another PE, you know that the receiving PE will not go beyond that blocking receive until the sending PE has completed its send. (The PVMFRECV routine, for instance, uses a blocking receive.) Yet synchronization is accomplished without involving the other PEs and is combined with the transfer of data. Further synchronization, such as using barriers, is usually not needed.

The follow-on to avoiding barriers is to avoid synchronization of any sort, if possible. Synchronization creates idle PEs, especially when some PEs have more work than others. Although synchronous communication may be easier to understand, asynchronous communication provides better performance.

Unlike some message-passing systems, PVM does not have an asynchronous receive (glossary, ), in which a receive is issued in parallel with the send, and the application later checks the status of this receive. Instead, it provides a nonblocking receive, the PVMFNRECV routine (see Section 2.7). On the CRAY T3E system, PVMFNRECV provides comparable performance to the PVMFRECV routine. If possible, write your code in such a way that it can use a nonblocking receive, so that if the message has not arrived, the code can do other work. See the example in the preceding section.