Petals Distribution

This question relates to many components of Petals, in particular the container and the CDK.

The main point is to be able to guarantee to a consumer that its message is handled as desired by the provider (and inversely for out messages and such).
The term reliability and persistence are used in more or less in the same way as in the JBI specification (annexe).

Many aspects exists:

1. When the container is not impacted by any problem, the JBI specification provides everything needed: if the message exchange pattern is followed (i.e. either DONE, FAULT or IN/OUT messages are exchanged as the pattern dictates) then both actors can assume everything is ok, and if an error is set it means something went wrong and the actors are notified with this information.
2. When the container has a problem (let's say crash), messages shouldn't be lost so that when it comes back, the execution can continue as desired.

With such guarantees, then the service consumers and providers can focus on their business and do not have to manage anything for handling such failures.
Of course in some corner case this may not be attainable, this has to be discussed here.

For each of these aspects, there is work to do:

1. We need to ascertain that for the whole path that an exchange take (i.e. from send to accept), errors are caught as needed and the exchange is correctly set in error and returned to the sender. This means that:
   • The CDK should send back an error if it drops messages (see PETALSCDK-135 and PETALSCDK-90).
   • We should verify everything in the container (issue to be created in case it is needed)
2. From the moment a send() call on the DeliveryChannel returns, the message should be considered as safely persisted until a call to accept() is done on the other side
   • We should take into account distributed scenario (i.e. persistence must happen both before the transport is involved on the sender side, and after on the receiver side). If possible in the local case, double persistence shouldn't happen.
   • Some choice should be made between these two options (or making that configurable... why not... I'm not even sure these questions are not already answered by the JBI specs):
     • the execution from "send()" to "the message is stored in the deliverychannel queue" is kind of atomic and only persisting in the DeliveryChannel queue (see PETALSESBCONT-339) is enough, if it doesn't happen the send method will throw an exception and the sender has to deal with it.
     • as soon as send is called, if something wrong happen before the message can be stored in the deliverychannel of the receiver (assuming it is persisted as in PETALSESBCONT-339), then when it's relevant, the message would have been persisted so that when the problem is solved it can be resent again. In that case send would return as soon as persistence has occurred. (not sure why this behaviour would be preferred... we should discuss that).
   • Some preliminary discussions on persistence in the delivery channel are in PETALSESBCONT-339 (which should maybe rejected and dispatched in multiple issues because it's a mess...)