All incoming files available are processed concurrently through a thread pool. Each file is processed by a task run by one thread. The task invokes the service provider asynchronously, and the thread returns to the pool without waiting the service provider response or acknowledgment. So we are able to process a lot of incoming files.
If the service provider stack is complex, we should avoid to invoke too many service providers concurrently otherwise it is possible to overloaded the system and time-outs will occur.
To avoid to process to many incoming files concurrently:
- we can use a sliding window mechanism whose the size is defined by the number of files to process concurrently,
- we wait the service provider response or acknowledgment to consider the incoming file as processed.
The sliding window is sized by 2 parameters at SU level:
- external-processor-block-size : The number of incoming files processed concurrently in the sliding window, or '0' for no limitation,
- external-processor-max-wait : The max waiting time that an incoming file waits to enter in the sliding window. If it can't, its processing will be postponed after the next polling period. This parameter has no sens if external-processor-block-size is set to '0'.
The existing thread pool is used to process incoming files simultaneously. As previously, it always returns to the pool just after to have invoke the service provider without waiting its response or ack. It is driven by following parameters at SU level:
- external-processor-thread-pool-size: the max size of the thread pool,
- external-processor-thread-pool-timeout: the max wait to get a thread from the pool. If not possible, the associated incoming file to process will be postponed after the next polling period.
Improvement Request
Resolved
Major
