main

2021/06/13

Python Collections library

Ref: https://www.geeksforgeeks.org/python-collections-module/

 1) Counters

It presents a stat of data

from collections import Counter

Counter({"A":5,"B":2})

array = [''cook","book","took","sook","book","book","aook"]

Counter(array)


2) OrderedDict

It remembers the order when key enters into the dict.

print("\nThis is an Ordered Dict:\n"
od = OrderedDict() 
od['a'] = 5
od['b'] = 2
od['c'] = 3
od['d'] = 4
    
for key, value in od.items(): 
    print(key, value)


3) DefaultDict

DefaultDict is also a sub-class to dictionary. It is used to provide some default values for the key that does not exist and never raises a KeyError.


4) ChainMap

ChainMap encapsulates many dictionaries into a single unit and returns a list of dictionaries.

from collections import ChainMap 
     
     
d1 = {'a': 1, 'b': 2}
d2 = {'c': 3, 'd': 4}
d3 = {'e': 5, 'f': 6}
  
# Defining the chainmap 
c = ChainMap(d1, d2, d3) 
     
print(c)

5)  NamedTuple

NamedTuple returns a tuple object with names for each position which the ordinary tuples lack. For example, consider a tuple names student where the first element represents fname, second represents lname and the third element represents the DOB. Suppose for calling fname instead of remembering the index position you can actually call the element by using the fname argument, then it will be really easy for accessing tuples element. This functionality is provided by the NamedTuple.

from collections import namedtuple
# Declaring namedtuple() 
Student = namedtuple('Student',['name','age','DOB']) 
    
# Adding values 
S = Student('Nandini','19','2541997'
    
# Access using index 
print ("The Student age using index is : ",end ="") 
print (S[1]) 
    
# Access using name  
print ("The Student name using keyname is : ",end ="") 
print (S.name)


6. Deque (pronounced deck)

Deque (Doubly Ended Queue) is the optimized list for quicker append and pop operations from both sides of the container. It provides O(1) time complexity for append and pop operations as compared to list with O(n) time complexity.

from collections import deque 
    
# initializing deque 
de = deque([1,2,3]) 
    
# using append() to insert element at right end  
# inserts 4 at the end of deque 
de.append(4
    
# printing modified deque 
print ("The deque after appending at right is : "
print (de) 
    
# using appendleft() to insert element at right end  
# inserts 6 at the beginning of deque 
de.appendleft(6
    
# printing modified deque 
print ("The deque after appending at left is : "

print (de)


7 UserDict

UserDict is a dictionary-like container that acts as a wrapper around the dictionary objects. This container is used when someone wants to create their own dictionary with some modified or new functionality. 

# Creating a Dictionary where 
# deletion is not allowed 
class MyDict(UserDict): 
        
    # Function to stop deleltion 
    # from dictionary 
    def __del__(self): 
        raise RuntimeError("Deletion not allowed"
            
    # Function to stop pop from  
    # dictionary 
    def pop(self, s = None): 
        raise RuntimeError("Deletion not allowed"
            
    # Function to stop popitem  
    # from Dictionary 
    def popitem(self, s = None): 
        raise RuntimeError("Deletion not allowed"
        
# Driver's code 
d = MyDict({'a':1
    'b': 2
    'c': 3})
    
d.pop(1)

8: UserList

9: UserString

2021/06/04

Ubuntu Upgrade from 16 to 18

 1) https://www.liquidweb.com/kb/how-to-upgrade-ubuntu-16-04-to-ubuntu-18-04/

However, sudo do-release-upgrade not working.


2) go to a second link

https://askubuntu.com/questions/1205332/do-release-upgrade-failing-on-ubuntu-16-04-lts-says-no-new-release-found


  1. Before, I had modified my default /etc/apt/sources.list file to the one described here: https://gist.github.com/rohitrawat/60a04e6ebe4a9ec1203eac3a11d4afc1 (that step was probably useless, since my sources.list file looked fine). After that, do-release-upgrade still produced a "No new release found" message.
  2. I changed xenial to bionic in /etc/apt/sources.list (there was nothing to change in the files under /etc/apt/sources.list.d/) by running this:

    $ sudo sed -i 's/xenial/bionic/g' /etc/apt/sources.list

  3. I upgraded everything with:

    $ sudo apt update && sudo apt -y dist-upgrade

  4. And reboot

2021/05/03

Cubkoo Sandbox Setup

Ref: https://medium.com/@seifreed/how-to-deploy-cuckoo-sandbox-431a6e65b848 

Ref 2: https://utopianknight.com/malware/cuckoo-installation-on-ubuntu-20/

Environment:  MacBook Pro 2019

VirtualBox 6.1

Set up Ubuntu 20 LTS

System Update

1) system update: sudo apt update && sudo apt upgrade -y

2) create a cuckoo user: sudo adduser cuckoo

3) add to sudo: sudo adduser cuckoo sudo

Deployment:

4) Add MongoDB support: sudo apt-key adv -keyserver keyserver.ubuntu.com -recv-keys 68818C72E52529D4

New curl -fsSL https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -


5) Add repo: echo “deb [arch=amd64] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/development multiverse” | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list

6) enable repo: sudo nano /etc/apt/sources.list

https://miro.medium.com/max/4800/1*u9Y0WNDzqtYJ-pivFUpXYA.png

7) sudo apt-update

8) package update again:

sudo apt install git mongodb-org-unstable python python-dev python-pip python-m2crypto libmagic1 swig libvirt-dev upx-ucl libssl-dev wget unzip p7zip-full geoip-database libgeoip-dev libjpeg-dev mono-utils ssdeep libfuzzy-dev exiftool curl openjdk-11-jre-headless xfce4 xfce4-goodies postgresql postgresql-contrib libpq-dev wkhtmltopdf xvfb xfonts-100dpi tcpdump libcap2-bin clamav clamav-daemon clamav-freshclam python-pil suricata libboost-all-dev qemu-kvm libvirt-clients libvirt-daemon virt-manager htop tmux gdebi-core tor privoxy libssl-dev libjansson-dev libmagic-dev automake apparmor-utils -y

9) pip update:

sudo -H pip install psycopg2 distorm3 pycrypto openpyxl

sudo -H pip install git+https://github.com/kbandla/pydeep.git

sudo -H pip install git+https://github.com/volatilityfoundation/volatility.git

sudo -H pip install pyopenssl -U

10) add our user to the KVM and libvirt group:

sudo usermod -a -G kvm $USER && sudo usermod -a -G libvirt $USER


11) enable package capture in VM:

sudo aa-disable /usr/sbin/tcpdump

sudo groupadd pcap

sudo usermod -a -G pcap cuckoo

sudo chgrp pcap /usr/sbin/tcpdump

sudo setcap cap_net_raw,cap_net_admin=eip /usr/sbin/tcpdump


12) sudo pip install cuckoo==2.0.6.2

 




2021/04/02

different data pipelines in data science projects

 There are many combinations which software stack you can choose for your data science projects. Here I am listing a few of them. Then I will create some examples to show how to glue them together.


1) redis, flower, celery, flask


2) elasticsearch/kibana, Jupiter for development, logstash, minio, nifi, rabbitmq, grafana, redis, faas, prometheus, alertmanager, flask


3) ES/Kibana, minio, nifi, rabbitmq, Casandra, hive, redis, MISP, keycloak, MongoDB,flask


4) LDAP, portioner, gâteau, drone, rabbitmq, kibana, minio, traefik



How to Supercharge Your Python Classes with Class Methods

  How to Supercharge Your Python Classes with Class Methods | by Siavash Yasini | May, 2024 | Towards Data Science As we just mentioned, a c...