celery and rabbitmq data pipeline example on mac OS

From  https://www.linode.com/docs/development/python/task-queue-celery-rabbitmq/

Install celery and rabbitmq
1) pip install celery
2) brew update
3) brew install rabbitmq
4) export PATH=$PATH:/usr/local/opt/rabbitmq/sbin

5) rabbitmq-server

   Then http://localhost:15672/ will give you access.

5. Celery application has two parts: 

• Workers that wait for messages from RabbitMQ and execute the tasks.
• Client that submit messages to RabbitMQ to trigger task execution, and eventually retrieve the result at a later time

6. Create a directory downloaderApp to hold our new python module, and a directory downloadedFiles where the downloaded files will be stored:

7. Create a downloaderApp.py module that will contain two functions, download and list, that will be the asynchronous tasks. 
  The following line is critical:

app = Celery('downloaderApp', backend='rpc://', broker='pyamqp://guest@localhost//')

This line creates:


A Celery application named downloaderApp


A broker on the localhost that will accept message via *Advanced Message Queuing Protocol (AMQP), the protocol used by RabbitMQ


A response backend where workers will store the return value of the task so that clients can retrieve it later (remember that task execution is asynchronous). If you omit backend, the task will still run, but the return value will be lost. rpc means the response will be sent to a RabbitMQ queue in a Remote Procedure Call pattern.

data pipeline, tutorial 2

Tutorial: Building An Analytics Data Pipeline In Python

https://www.dataquest.io/blog/data-pipelines-tutorial/

data pipeline tutorial

How to Set Up a Task Queue with Celery and RabbitMQ

https://www.linode.com/docs/development/python/task-queue-celery-rabbitmq/