Petals CDK

Rework the Acceptor/Processor thread pools

Details

  • Type: Task Task
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 5.4.2
  • Fix Version/s: 5.6.0
  • Component/s: runtime
  • Security Level: Public
  • Description:
    Hide

    The acceptor thread pool is a fixed size list of threads. Each thread accepts messages from the delivery channel before to give them to the message processors.

    In relation with PETALSCDK-135, the goal is to rewrite the Acceptor/Processor message handling.

    The final result should be something like this:

    • Acceptor threads that dies are recreated
    • There is only one acceptor thread by default (there should be no need for more, but who knows... so we keep the pool)
    • There is a pool for the processor threads (and we remove the pool of processors) (and the CDK parameter message-processor-max-pool-size is removed)
    • Acceptor that are waiting in retry stop waiting as soon as a new processor is available
    Show
    The acceptor thread pool is a fixed size list of threads. Each thread accepts messages from the delivery channel before to give them to the message processors. In relation with PETALSCDK-135, the goal is to rewrite the Acceptor/Processor message handling. The final result should be something like this:
    • Acceptor threads that dies are recreated
    • There is only one acceptor thread by default (there should be no need for more, but who knows... so we keep the pool)
    • There is a pool for the processor threads (and we remove the pool of processors) (and the CDK parameter message-processor-max-pool-size is removed)
    • Acceptor that are waiting in retry stop waiting as soon as a new processor is available
  • Environment:
    -

Issue Links

Depends

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Victor NOËL added a comment - Mon, 15 Jun 2015 - 16:04:08 +0200

Question for you Christophe: are the classes in org.ow2.petals.component.framework.monitoring.defect only used to notify in case there is no thread available in the acceptor thread pool?

If the acceptor thread pool is refactored and we can be sure there always be thread available, can we remove the classes in org.ow2.petals.component.framework.monitoring.defect?

Thanks

Show
Victor NOËL added a comment - Mon, 15 Jun 2015 - 16:04:08 +0200 Question for you Christophe: are the classes in org.ow2.petals.component.framework.monitoring.defect only used to notify in case there is no thread available in the acceptor thread pool? If the acceptor thread pool is refactored and we can be sure there always be thread available, can we remove the classes in org.ow2.petals.component.framework.monitoring.defect? Thanks
Hide
Permalink
Christophe DENEUX added a comment - Mon, 15 Jun 2015 - 16:14:00 +0200

Yes, you can remove the defect if it has no more sens. Don't forget to update user documentation.

Show
Christophe DENEUX added a comment - Mon, 15 Jun 2015 - 16:14:00 +0200 Yes, you can remove the defect if it has no more sens. Don't forget to update user documentation.
Hide
Permalink
Victor NOËL added a comment - Mon, 15 Jun 2015 - 16:49:56 +0200

And can you remind me the rationale behind having two sets of threads: one for accepting messages and one for processing them?

Thanks

Show
Victor NOËL added a comment - Mon, 15 Jun 2015 - 16:49:56 +0200 And can you remind me the rationale behind having two sets of threads: one for accepting messages and one for processing them? Thanks
Hide
Permalink
Christophe DENEUX added a comment - Tue, 16 Jun 2015 - 17:53:57 +0200

The mechanism based on acceptors/workers was written because of:

  • DeleveryChannel.accept() is a blocking API,
  • to limit resource consumption when there is no message to process (only few threads are running),
  • to limit thread creations (a thread will process several messages until no message is available during a given duration)
Show
Christophe DENEUX added a comment - Tue, 16 Jun 2015 - 17:53:57 +0200 The mechanism based on acceptors/workers was written because of:
  • DeleveryChannel.accept() is a blocking API,
  • to limit resource consumption when there is no message to process (only few threads are running),
  • to limit thread creations (a thread will process several messages until no message is available during a given duration)
Hide
Permalink
Victor NOËL added a comment - Wed, 17 Jun 2015 - 09:24:13 +0200

Ok, but why didn't we get just a pool of threads that accept and then directly process? I don't see how it is different… is it?

One difference I see is the fact that acceptor threads wait if there is no available processor threads, and then they drop the messages!
Does it mean we have as a requirement : messages must be dropped if there is no resources available to process them?

I'm trying to clarify the requirements we are trying to achieve independently of the current implementation that seems to have been written by many people at different times

Show
Victor NOËL added a comment - Wed, 17 Jun 2015 - 09:24:13 +0200 Ok, but why didn't we get just a pool of threads that accept and then directly process? I don't see how it is different… is it? One difference I see is the fact that acceptor threads wait if there is no available processor threads, and then they drop the messages! Does it mean we have as a requirement : messages must be dropped if there is no resources available to process them? I'm trying to clarify the requirements we are trying to achieve independently of the current implementation that seems to have been written by many people at different times
Hide
Permalink
Christophe DENEUX added a comment - Wed, 17 Jun 2015 - 09:55:39 +0200 - edited

If you are able to use only a thread pool to accept and process message, and taking account the resource constraints, it's ok for me. Your thread pool should be sized from 10 to 50 by default as the current message processor thread pool. In my mind, you should update the current message processor thread pool to avoid too changes for our users (configuration, monitoring)

When no resource is available it should be better to return an error according to the MEP. Dropping message was an easy way but it can be problematic if no timeout is set by the caller: it will wait indefinitely a reply or ack.

Show
Christophe DENEUX added a comment - Wed, 17 Jun 2015 - 09:55:39 +0200 - edited If you are able to use only a thread pool to accept and process message, and taking account the resource constraints, it's ok for me. Your thread pool should be sized from 10 to 50 by default as the current message processor thread pool. In my mind, you should update the current message processor thread pool to avoid too changes for our users (configuration, monitoring) When no resource is available it should be better to return an error according to the MEP. Dropping message was an easy way but it can be problematic if no timeout is set by the caller: it will wait indefinitely a reply or ack.
Hide
Permalink
Victor NOËL added a comment - Wed, 17 Jun 2015 - 10:11:57 +0200

Ok,

and I agree with the second point, let's discuss that during weekly meeting since it's not straightforward.

Show
Victor NOËL added a comment - Wed, 17 Jun 2015 - 10:11:57 +0200 Ok, and I agree with the second point, let's discuss that during weekly meeting since it's not straightforward.
Hide
Permalink
Christophe DENEUX added a comment - Mon, 5 Oct 2015 - 15:42:53 +0200

Postponed to 5.5.1 or upper

Show
Christophe DENEUX added a comment - Mon, 5 Oct 2015 - 15:42:53 +0200 Postponed to 5.5.1 or upper
Hide
Permalink
Victor NOËL added a comment - Thu, 11 Feb 2016 - 15:24:53 +0100

Updated the description.

Show
Victor NOËL added a comment - Thu, 11 Feb 2016 - 15:24:53 +0100 Updated the description.

People

Dates

  • Created:
    Mon, 16 Mar 2015 - 11:45:20 +0100
    Updated:
    Mon, 15 Feb 2016 - 10:42:19 +0100
    Resolved:
    Mon, 15 Feb 2016 - 10:42:19 +0100
Atlassian JIRA (v4.1.2#531) | Report a problem | Request a feature | Contact administrators
Powered by a free Atlassian JIRA open source license for Petals ESB. Try JIRA - bug tracking software for your team.