three message brokers: redis, kafaka and RabbitMQ

There are three popular asynchronous message brokers: Redis, Kafaka and RabbitMQ. When we want to handle asynchronous message, we use message broker to handle the data. The message broker is kind of post office. Somebody drops the mail(message) in the office. Post office will deliver those mails based on the mailing address(routing) to its receipt. 

In a Synchronous communication, the caller waits for a response before sending the next message, and it operates as a REST protocol on top of HTTP. On the contrary, in an Asynchronous communication the messages are sent without waiting for a response. This is suited for distributed systems, and usually requires a message broker to manage the messages.

There are three parameters to consider: broker scale, data persistency and consumer capability. 

RabbitMQ: 

Scale: based on configuration and resources, the ballpark here is around 50K msg per second.

Persistency: both persistent and transient messages are supported.

One-to-one vs one-to-many consumers: both.

Kafaka:

Scale: can send up to a millions messages per second.

Persistency: yes.

One-to-one vs one-to-many consumers: only one-to-many (seems strange at first glance, right?!).

Redis:

Scale: can send up to a million messages per second.

Persistency: basically, no – it’s an in-memory datastore.

One-to-one vs one-to-many consumers: both.

Some Information above is take from the following article:

https://otonomo.io/blog/redis-kafka-or-rabbitmq-which-microservices-message-broker-to-choose/

A Guide to Python's Magic Methods

 https://rszalski.github.io/magicmethods/

__init__

__repr__

__getattr__

__setattr__

__call__(self, [args...])







context manager

class Closer:
    '''A context manager to automatically close an object with a close method
    in a with statement.'''

    def __init__(self, obj):
        self.obj = obj

    def __enter__(self):
        return self.obj # bound to target

    def __exit__(self, exception_type, exception_val, trace):
        try:
           self.obj.close()
        except AttributeError: # obj isn't closable
           print 'Not closable.'
           return True # exception handled successfully

__copy__(self)
__deepcopy__(self, memodict={})

Special Python Class methods

 class MyClass:

    def method(self):
        return 'instance method called', self

    @classmethod
    def classmethod(cls):
        return 'class method called', cls

    @staticmethod
    def staticmethod():
        return 'static method called'

Instead of accepting a self parameter, class methods take a cls parameter that points to the class—and not the object instance—when the method is called.
Because the class method only has access to this cls argument, it can’t modify object instance state. That would require access to self. However, class methods can still modify class state that applies across all instances of the class.
Static Methods
The third method, MyClass.staticmethod was marked with a @staticmethod decorator to flag it as a static method.
This type of method takes neither a self nor a cls parameter (but of course it’s free to accept an arbitrary number of other parameters).
Therefore a static method can neither modify object state nor class state. Static methods are restricted in what data they can access - and they’re primarily a way to namespace your methods.

Asynchronous Tasks Using Flask, Redis, and Celery

Use  Celery

Celery is an asynchronous task queue based on distributed message passing to distribute workload across machines or threads. A celery system consists of a client, a broker, and several workers. 

These workers are responsible for the execution of the tasks or pieces of work that are placed in the queue and relaying the results. With Celery, you can have both local and remote workers meaning that work can be delegated to different and more capable machines over the internet and results relayed back to the client.

This way, the load on the main machine is alleviated and more resources are available to handle user requests as they come in.

The client in a Celery setup is responsible for issuing jobs to the workers and also communicating with them using a message broker. The broker facilitates the communication between the client and the workers in a Celery installation through a message queue, where a message is added to the queue and the broker delivers it to the client.

Examples of such message brokers include Redis and RabbitMQ.

How to set up Redis:

https://redis.io/topics/quickstart

https://stackabuse.com/asynchronous-tasks-using-flask-redis-and-celery/#disqus_thread

using flower to track the data flow

https://github.com/mher/flower

celery flower -A proj --address=127.0.0.1 --port=5555

step by step with Prometheus,Grafana, Node_Exporter with Docker

 https://medium.com/skedler/monitoring-servers-and-docker-containers-using-prometheus-with-grafana-87cf961fe1e0

https://royportas.com/posts/2020-02-25-prometheus-and-grafana-configuration-for-docker-compose/

https://support.skedler.com/support/solutions/articles/8000075591-skedler-reports-docker-compose-installation-guide

https://linoxide.com/containers/setup-monitoring-docker-containers-prometheus/

The following graph is very helpful to understand the whole structure.