Details
-
Type:
Improvement Request
-
Status:
Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 3.3.3
-
Fix Version/s: 4.0.0-beta-1, 4.0.0
-
Component/s: File listening (consumer part)
-
Security Level: Public
-
- Environment:
- -
Activity
| Field | Original Value | New Value |
|---|---|---|
| Status | New [ 10000 ] | Open [ 10002 ] |
| Priority | Major [ 3 ] |
| Attachment | PETALSBCFTRANS-62.patch [ 10790 ] |
| Summary | The processing of incoming files by block does not work as expected | Introduce a sliding window to process incoming files |
| Issue Type | Bug [ 1 ] | Improvement Request [ 4 ] |
| Affects Version/s | 3.3.3 [ 10679 ] | |
| Affects Version/s | 3.3.2 [ 10643 ] | |
| Description |
Incoming files are processed by block using a thread pool that execute one task by file. Limiting the max size of the thread pool can limit the number of files processed concurrently.
Each task executed by the thread pool should send the request to the service provider and wait its reply or acknowledge, but the request is sent asynchronously and the reply or ack is not waited. So the task is ended too early and the processing by block has no real effect to limit the number of file processed concurrently. Please, use synchronous way to invoke the request. |
All incoming files available are processed concurrently through a thread pool. Each file is processed by a task run by one thread. The task invokes the service provider asynchronously, and the thread returns to the pool without waiting the service provider response or acknowledgment. So we are able to process a lot of incoming files.
If the service provider stack is complex, we should avoid to invoke too many service providers concurrently otherwise it is possible to overloaded the system and time-outs will occur. To avoid to process to many incoming files concurrently: * we can use a sliding window mechanism whose the size is defined by the number of files to process concurrently, * we wait the service provider response or acknowledgment to consider the incoming file as processed. The sliding window is sized by 2 parameters at SU level: * {{external-processor-block-size}} : The number of incoming files processed concurrently in the sliding window, or '0' for no limitation, * {{external-processor-max-wait}} : The max waiting time that an incoming file waits to enter in the sliding window. If it can't, its processing will be postponed after the next polling period. This parameter has no sens if {{external-processor-block-size}} is set to '0'. The existing thread pool is used to process incoming files simultaneously. As previously, it always returns to the pool just after to have invoke the service provider without waiting its response or ack. It is driven by following parameters: * {{external-processor-thread-pool-size}}: the max size of the thread pool, * {{external-processor-thread-pool-timeout}}: the max wait to get a thread from the pool. If not possible, the associated incoming file to process will be postponed after the next polling period. |
| Description |
All incoming files available are processed concurrently through a thread pool. Each file is processed by a task run by one thread. The task invokes the service provider asynchronously, and the thread returns to the pool without waiting the service provider response or acknowledgment. So we are able to process a lot of incoming files.
If the service provider stack is complex, we should avoid to invoke too many service providers concurrently otherwise it is possible to overloaded the system and time-outs will occur. To avoid to process to many incoming files concurrently: * we can use a sliding window mechanism whose the size is defined by the number of files to process concurrently, * we wait the service provider response or acknowledgment to consider the incoming file as processed. The sliding window is sized by 2 parameters at SU level: * {{external-processor-block-size}} : The number of incoming files processed concurrently in the sliding window, or '0' for no limitation, * {{external-processor-max-wait}} : The max waiting time that an incoming file waits to enter in the sliding window. If it can't, its processing will be postponed after the next polling period. This parameter has no sens if {{external-processor-block-size}} is set to '0'. The existing thread pool is used to process incoming files simultaneously. As previously, it always returns to the pool just after to have invoke the service provider without waiting its response or ack. It is driven by following parameters: * {{external-processor-thread-pool-size}}: the max size of the thread pool, * {{external-processor-thread-pool-timeout}}: the max wait to get a thread from the pool. If not possible, the associated incoming file to process will be postponed after the next polling period. |
All incoming files available are processed concurrently through a thread pool. Each file is processed by a task run by one thread. The task invokes the service provider asynchronously, and the thread returns to the pool without waiting the service provider response or acknowledgment. So we are able to process a lot of incoming files.
If the service provider stack is complex, we should avoid to invoke too many service providers concurrently otherwise it is possible to overloaded the system and time-outs will occur. To avoid to process to many incoming files concurrently: * we can use a sliding window mechanism whose the size is defined by the number of files to process concurrently, * we wait the service provider response or acknowledgment to consider the incoming file as processed. The sliding window is sized by 2 parameters at SU level: * {{external-processor-block-size}} : The number of incoming files processed concurrently in the sliding window, or '0' for no limitation, * {{external-processor-max-wait}} : The max waiting time that an incoming file waits to enter in the sliding window. If it can't, its processing will be postponed after the next polling period. This parameter has no sens if {{external-processor-block-size}} is set to '0'. The existing thread pool is used to process incoming files simultaneously. As previously, it always returns to the pool just after to have invoke the service provider without waiting its response or ack. It is driven by following parameters at SU level: * {{external-processor-thread-pool-size}}: the max size of the thread pool, * {{external-processor-thread-pool-timeout}}: the max wait to get a thread from the pool. If not possible, the associated incoming file to process will be postponed after the next polling period. |
| Status | Open [ 10002 ] | In Progress [ 10003 ] |
| Status | In Progress [ 10003 ] | Resolved [ 10004 ] |
| Fix Version/s | 4.0.0-beta-1 [ 10767 ] | |
| Resolution | Fixed [ 1 ] |
| Status | Resolved [ 10004 ] | Open [ 10002 ] |
| Resolution | Fixed [ 1 ] |
| Fix Version/s | 4.0.0 [ 10810 ] |
| Status | Open [ 10002 ] | In Progress [ 10003 ] |
| Status | In Progress [ 10003 ] | Resolved [ 10004 ] |
| Resolution | Fixed [ 1 ] |
| Transition | Status Change Time | Execution Times | Last Executer | Last Execution Date | |||||||||
|
|
|
|
|
|||||||||
|
|
|
|
|
|||||||||
|
|
|
|
|
|||||||||
|
|
|
|
|
|||||||||
|
|
|
|
|
|||||||||
|
|
|
|
|
Potential patch