Dynamically manage pools of asyncio tasks
Go to file
2022-03-29 19:34:27 +02:00
docs renamed group_size to num_concurrent in _map 2022-03-29 19:34:27 +02:00
requirements sphinx documentation; adjusted all docstrings; moved some modules to non-public subpackage 2022-03-24 13:38:30 +01:00
src/asyncio_taskpool renamed group_size to num_concurrent in _map 2022-03-29 19:34:27 +02:00
tests renamed group_size to num_concurrent in _map 2022-03-29 19:34:27 +02:00
usage renamed group_size to num_concurrent in _map 2022-03-29 19:34:27 +02:00
.coveragerc skipping unix socket tests on Windows 2022-02-25 21:17:14 +01:00
.gitignore sphinx documentation; adjusted all docstrings; moved some modules to non-public subpackage 2022-03-24 13:38:30 +01:00
COPYING license & copyright notices; docstrings for each module; extended readme 2022-02-09 23:14:42 +01:00
COPYING.LESSER license & copyright notices; docstrings for each module; extended readme 2022-02-09 23:14:42 +01:00
coverage.sh Initial commit 2022-02-03 20:06:30 +01:00
pyproject.toml Initial commit 2022-02-03 20:06:30 +01:00
README.md improved and extended usage/readme docs; small fixes 2022-03-14 18:09:30 +01:00
setup.cfg sphinx documentation; adjusted all docstrings; moved some modules to non-public subpackage 2022-03-24 13:38:30 +01:00

asyncio-taskpool

Dynamically manage pools of asyncio tasks

Contents

Summary

A task pool is an object with a simple interface for aggregating and dynamically managing asynchronous tasks.

With an interface that is intentionally similar to the multiprocessing.Pool class from the standard library, the TaskPool provides you such methods as apply, map, and starmap to execute coroutines concurrently as asyncio.Task objects. There is no limitation imposed on what kind of tasks can be run or in what combination, when new ones can be added, or when they can be cancelled.

For a more streamlined use-case, the SimpleTaskPool provides an even more intuitive and simple interface at the cost of flexibility.

If you need control over a task pool at runtime, you can launch an asynchronous ControlServer to be able to interface with the pool from an outside process or via a network, and stop/start tasks within the pool as you wish.

Usage

Generally speaking, a task is added to a pool by providing it with a coroutine function reference as well as the arguments for that function. Here is what that could look like in the most simplified form:

from asyncio_taskpool import SimpleTaskPool

...


async def work(_foo, _bar): ...


...


async def main():
    pool = SimpleTaskPool(work, args=('xyz', 420))
    await pool.start(5)
    ...
    pool.stop(3)
    ...
    pool.lock()
    await pool.gather_and_close()
    ...

Since one of the main goals of asyncio-taskpool is to be able to start/stop tasks dynamically or "on-the-fly", most of the associated methods are non-blocking most of the time. A notable exception is the gather_and_close method for awaiting the return of all tasks in the pool. (It is essentially a glorified wrapper around the asyncio.gather function.)

For working and fully documented demo scripts see USAGE.md.

Installation

pip install asyncio-taskpool

Dependencies

Python Version 3.8+, tested on Linux

Testing

Install asyncio-taskpool[dev] dependencies or just manually install coverage with pip. Execute the ./coverage.sh shell script to run all unit tests and receive the coverage report.

License

asyncio-taskpool is licensed under the GNU LGPL version 3.0 specifically.

The full license texts for the GNU GPLv3.0 and the GNU LGPLv3.0 are included in this repository. If not, see https://www.gnu.org/licenses/.


© 2022 Daniil Fajnberg