Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upLimiting Concurrency #160
Limiting Concurrency #160
Comments
|
Hi @jlaxson , I think in your case you can just use one |
|
Hi, thanks for your quick reply.
I had looked a bit through the code but all I see is Executor::_spawn which is always used to spawn a constant number of threads. Can you point me towards more detail?
I think this would mean that I need to know (or control) a priori the order of all the heavy_tasks? If I don't know which heavy_task would become ready first, is there a way to accomplish this? |
|
@jlaxson Hi, we do spawn a fixed number of threads but they transition back and forth to sleep and awake depending on the present task parallelism. You may check Secondly, Yes, you need to know the dependency. However, you may check Thanks again for the discussion. |
Hi - I really like what you've built here. I have a use case that I can't quite figure out how to map to cpp-taskflow, hoping I could get some advice.
In my workflow I have a number of steps that I'd like to artificially limit concurrency - for instance, a subflow that consumes a very large amount of memory. Or that consume limited external resources (gpus, etc). Or even to differentiate between CPU-bound tasks (at most one worker per cpu thread) and IO bound tasks (perhaps many more than one per thread?).
I think I could emulate this with mutexes or other synchronization outside of taskflow itself, but that has the side effect of consuming/blocking workers when they could be working on other ready tasks.
One potential solution would be to have the ability to assign tasks/subflows to specific executors, with dependencies across the executors. So I could have my GPU executor, with concurrency 2, in parallel to the CPU executor with native hardware concurrency. Syntax could look something like:
Wondering if there's a way to achieve this with the functionality you have now? Otherwise, a humble feature suggestion.