Dynamically manage pools of asyncio tasks
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
Daniil Fajnberg 73aa93a9b7
Fix unit tests
5 months ago
.github/workflows Fix cancel message bug for Python 3.8; test coverage workaround for Python version conditions 6 months ago
docs Fix control server response writes 5 months ago
requirements sphinx documentation; adjusted all docstrings; moved some modules to non-public subpackage 6 months ago
src/asyncio_taskpool Fix control server response writes 5 months ago
tests Fix unit tests 5 months ago
usage made `start` "non-async" using meta task 6 months ago
.coveragerc Fix cancel message bug for Python 3.8; test coverage workaround for Python version conditions 6 months ago
.gitignore sphinx documentation; adjusted all docstrings; moved some modules to non-public subpackage 6 months ago
.readthedocs.yaml Fixed Python 3.8 compatibility bugs; classmethod+property workaround; control session buffer 6 months ago
COPYING license & copyright notices; docstrings for each module; extended readme 8 months ago
COPYING.LESSER license & copyright notices; docstrings for each module; extended readme 8 months ago
README.md Fixed Python 3.8 compatibility bugs; classmethod+property workaround; control session buffer 6 months ago
cloc.sh Add workflows & badges 6 months ago
coverage.sh Optimize coverage script/settings 6 months ago
pyproject.toml Initial commit 8 months ago
setup.cfg Fix control server response writes 5 months ago

README.md

asyncio-taskpool

GitHub last commit Lines of code Lines of comments Test coverage License: LGPL v3.0 PyPI version

Dynamically manage pools of asyncio tasks

Full documentation available at RtD.


Contents

Summary

A task pool is an object with a simple interface for aggregating and dynamically managing asynchronous tasks.

With an interface that is intentionally similar to the multiprocessing.Pool class from the standard library, the TaskPool provides you such methods as apply, map, and starmap to execute coroutines concurrently as asyncio.Task objects. There is no limitation imposed on what kind of tasks can be run or in what combination, when new ones can be added, or when they can be cancelled.

For a more streamlined use-case, the SimpleTaskPool provides an even more intuitive and simple interface at the cost of flexibility.

If you need control over a task pool at runtime, you can launch an asynchronous ControlServer to be able to interface with the pool from an outside process or via a network, and stop/start tasks within the pool as you wish.

Usage

Generally speaking, a task is added to a pool by providing it with a coroutine function reference as well as the arguments for that function. Here is what that could look like in the most simplified form:

from asyncio_taskpool import SimpleTaskPool
...
async def work(_foo, _bar): ...

async def main():
    pool = SimpleTaskPool(work, args=('xyz', 420))
    pool.start(5)
    ...
    pool.stop(3)
    ...
    await pool.gather_and_close()

Since one of the main goals of asyncio-taskpool is to be able to start/stop tasks dynamically or "on-the-fly", most of the associated methods are non-blocking most of the time. A notable exception is the gather_and_close method for awaiting the return of all tasks in the pool. (It is essentially a glorified wrapper around the asyncio.gather function.)

For working and fully documented demo scripts see USAGE.md.

Installation

pip install asyncio-taskpool

Dependencies

Python Version 3.8+, tested on Linux

Testing

Install coverage with pip, then execute the ./coverage.sh shell script to run all unit tests and save the coverage report.

License

asyncio-taskpool is licensed under the GNU LGPL version 3.0 specifically.

The full license texts for the GNU GPLv3.0 and the GNU LGPLv3.0 are included in this repository. If not, see https://www.gnu.org/licenses/.


© 2022 Daniil Fajnberg