Asynchronous Tasks Using Flask, Redis, and Celery

Use  Celery

Celery is an asynchronous task queue based on distributed message passing to distribute workload across machines or threads. A celery system consists of a client, a broker, and several workers. 

These workers are responsible for the execution of the tasks or pieces of work that are placed in the queue and relaying the results. With Celery, you can have both local and remote workers meaning that work can be delegated to different and more capable machines over the internet and results relayed back to the client.

This way, the load on the main machine is alleviated and more resources are available to handle user requests as they come in.

The client in a Celery setup is responsible for issuing jobs to the workers and also communicating with them using a message broker. The broker facilitates the communication between the client and the workers in a Celery installation through a message queue, where a message is added to the queue and the broker delivers it to the client.

Examples of such message brokers include Redis and RabbitMQ.

How to set up Redis:

https://redis.io/topics/quickstart

https://stackabuse.com/asynchronous-tasks-using-flask-redis-and-celery/#disqus_thread

using flower to track the data flow

https://github.com/mher/flower

celery flower -A proj --address=127.0.0.1 --port=5555