Ubuntu Upgrade from 16 to 18

 1) https://www.liquidweb.com/kb/how-to-upgrade-ubuntu-16-04-to-ubuntu-18-04/

However, sudo do-release-upgrade not working.

2) go to a second link

https://askubuntu.com/questions/1205332/do-release-upgrade-failing-on-ubuntu-16-04-lts-says-no-new-release-found

1. Before, I had modified my default /etc/apt/sources.list file to the one described here: https://gist.github.com/rohitrawat/60a04e6ebe4a9ec1203eac3a11d4afc1 (that step was probably useless, since my sources.list file looked fine). After that, do-release-upgrade still produced a "No new release found" message.

2. I changed xenial to bionic in /etc/apt/sources.list (there was nothing to change in the files under /etc/apt/sources.list.d/) by running this:

   $ sudo sed -i 's/xenial/bionic/g' /etc/apt/sources.list

3. I upgraded everything with:

   $ sudo apt update && sudo apt -y dist-upgrade

4. And reboot

Cubkoo Sandbox Setup

Ref: https://medium.com/@seifreed/how-to-deploy-cuckoo-sandbox-431a6e65b848 

Ref 2: https://utopianknight.com/malware/cuckoo-installation-on-ubuntu-20/

Environment:  MacBook Pro 2019

VirtualBox 6.1

Set up Ubuntu 20 LTS

System Update

1) system update: sudo apt update && sudo apt upgrade -y

2) create a cuckoo user: sudo adduser cuckoo

3) add to sudo: sudo adduser cuckoo sudo

Deployment:

4) Add MongoDB support: sudo apt-key adv -keyserver keyserver.ubuntu.com -recv-keys 68818C72E52529D4

New curl -fsSL https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -

5) Add repo: echo “deb [arch=amd64] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/development multiverse” | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list

6) enable repo: sudo nano /etc/apt/sources.list

https://miro.medium.com/max/4800/1*u9Y0WNDzqtYJ-pivFUpXYA.png

7) sudo apt-update

8) package update again:

sudo apt install git mongodb-org-unstable python python-dev python-pip python-m2crypto libmagic1 swig libvirt-dev upx-ucl libssl-dev wget unzip p7zip-full geoip-database libgeoip-dev libjpeg-dev mono-utils ssdeep libfuzzy-dev exiftool curl openjdk-11-jre-headless xfce4 xfce4-goodies postgresql postgresql-contrib libpq-dev wkhtmltopdf xvfb xfonts-100dpi tcpdump libcap2-bin clamav clamav-daemon clamav-freshclam python-pil suricata libboost-all-dev qemu-kvm libvirt-clients libvirt-daemon virt-manager htop tmux gdebi-core tor privoxy libssl-dev libjansson-dev libmagic-dev automake apparmor-utils -y

9) pip update:

sudo -H pip install psycopg2 distorm3 pycrypto openpyxl

sudo -H pip install git+https://github.com/kbandla/pydeep.git

sudo -H pip install git+https://github.com/volatilityfoundation/volatility.git

sudo -H pip install pyopenssl -U

10) add our user to the KVM and libvirt group:

sudo usermod -a -G kvm $USER && sudo usermod -a -G libvirt $USER

11) enable package capture in VM:

sudo aa-disable /usr/sbin/tcpdump

sudo groupadd pcap

sudo usermod -a -G pcap cuckoo

sudo chgrp pcap /usr/sbin/tcpdump

sudo setcap cap_net_raw,cap_net_admin=eip /usr/sbin/tcpdump

12) sudo pip install cuckoo==2.0.6.2

 

different data pipelines in data science projects

 There are many combinations which software stack you can choose for your data science projects. Here I am listing a few of them. Then I will create some examples to show how to glue them together.

1) redis, flower, celery, flask

2) elasticsearch/kibana, Jupiter for development, logstash, minio, nifi, rabbitmq, grafana, redis, faas, prometheus, alertmanager, flask

3) ES/Kibana, minio, nifi, rabbitmq, Casandra, hive, redis, MISP, keycloak, MongoDB,flask

4) LDAP, portioner, gâteau, drone, rabbitmq, kibana, minio, traefik

three message brokers: redis, kafaka and RabbitMQ

There are three popular asynchronous message brokers: Redis, Kafaka and RabbitMQ. When we want to handle asynchronous message, we use message broker to handle the data. The message broker is kind of post office. Somebody drops the mail(message) in the office. Post office will deliver those mails based on the mailing address(routing) to its receipt. 

In a Synchronous communication, the caller waits for a response before sending the next message, and it operates as a REST protocol on top of HTTP. On the contrary, in an Asynchronous communication the messages are sent without waiting for a response. This is suited for distributed systems, and usually requires a message broker to manage the messages.

There are three parameters to consider: broker scale, data persistency and consumer capability. 

RabbitMQ: 

Scale: based on configuration and resources, the ballpark here is around 50K msg per second.

Persistency: both persistent and transient messages are supported.

One-to-one vs one-to-many consumers: both.

Kafaka:

Scale: can send up to a millions messages per second.

Persistency: yes.

One-to-one vs one-to-many consumers: only one-to-many (seems strange at first glance, right?!).

Redis:

Scale: can send up to a million messages per second.

Persistency: basically, no – it’s an in-memory datastore.

One-to-one vs one-to-many consumers: both.

Some Information above is take from the following article:

https://otonomo.io/blog/redis-kafka-or-rabbitmq-which-microservices-message-broker-to-choose/

A Guide to Python's Magic Methods

 https://rszalski.github.io/magicmethods/

__init__

__repr__

__getattr__

__setattr__

__call__(self, [args...])







context manager

class Closer:
    '''A context manager to automatically close an object with a close method
    in a with statement.'''

    def __init__(self, obj):
        self.obj = obj

    def __enter__(self):
        return self.obj # bound to target

    def __exit__(self, exception_type, exception_val, trace):
        try:
           self.obj.close()
        except AttributeError: # obj isn't closable
           print 'Not closable.'
           return True # exception handled successfully

__copy__(self)
__deepcopy__(self, memodict={})

Special Python Class methods

 class MyClass:

    def method(self):
        return 'instance method called', self

    @classmethod
    def classmethod(cls):
        return 'class method called', cls

    @staticmethod
    def staticmethod():
        return 'static method called'

Instead of accepting a self parameter, class methods take a cls parameter that points to the class—and not the object instance—when the method is called.
Because the class method only has access to this cls argument, it can’t modify object instance state. That would require access to self. However, class methods can still modify class state that applies across all instances of the class.
Static Methods
The third method, MyClass.staticmethod was marked with a @staticmethod decorator to flag it as a static method.
This type of method takes neither a self nor a cls parameter (but of course it’s free to accept an arbitrary number of other parameters).
Therefore a static method can neither modify object state nor class state. Static methods are restricted in what data they can access - and they’re primarily a way to namespace your methods.

Asynchronous Tasks Using Flask, Redis, and Celery

Use  Celery

Celery is an asynchronous task queue based on distributed message passing to distribute workload across machines or threads. A celery system consists of a client, a broker, and several workers. 

These workers are responsible for the execution of the tasks or pieces of work that are placed in the queue and relaying the results. With Celery, you can have both local and remote workers meaning that work can be delegated to different and more capable machines over the internet and results relayed back to the client.

This way, the load on the main machine is alleviated and more resources are available to handle user requests as they come in.

The client in a Celery setup is responsible for issuing jobs to the workers and also communicating with them using a message broker. The broker facilitates the communication between the client and the workers in a Celery installation through a message queue, where a message is added to the queue and the broker delivers it to the client.

Examples of such message brokers include Redis and RabbitMQ.

How to set up Redis:

https://redis.io/topics/quickstart

https://stackabuse.com/asynchronous-tasks-using-flask-redis-and-celery/#disqus_thread

using flower to track the data flow

https://github.com/mher/flower

celery flower -A proj --address=127.0.0.1 --port=5555