hands-on with Kubernetes

https://kubernetes.io/blog/2019/07/23/get-started-with-kubernetes-using-python/

• A Kubernetes service - I’m using Docker Desktop with Kubernetes in this walkthrough, but you can use one of the others. See Getting Started for a full listing.
• Python 3.7 installed
• Git installed

FROM python:3.7

RUN mkdir /app
WORKDIR /app
ADD . /app/
RUN pip install -r requirements.txt

EXPOSE 5000
CMD ["python", "/app/main.py"]

1. Get the official Python Base Image for version 3.7 from Docker Hub.
2. In the image, create a directory named app.
3. Set the working directory to that new app directory.
4. Copy the local directory’s contents to that new folder into the image.
5. Run the pip installer (just like we did earlier) to pull the requirements into the image.
6. Inform Docker the container listens on port 5000.
7. Configure the starting command to use when the container starts.

docker build -f Dockerfile -t hello-python:latest .

docker run -p 5001:5000 hello-python

Running in Kubernetes

install kubernetes,


brew install kubectl 

https://kubernetes.io/docs/tasks/tools/install-kubectl/

celery and rabbitmq data pipeline example on mac OS

From  https://www.linode.com/docs/development/python/task-queue-celery-rabbitmq/

Install celery and rabbitmq
1) pip install celery
2) brew update
3) brew install rabbitmq
4) export PATH=$PATH:/usr/local/opt/rabbitmq/sbin

5) rabbitmq-server

   Then http://localhost:15672/ will give you access.

5. Celery application has two parts: 

• Workers that wait for messages from RabbitMQ and execute the tasks.
• Client that submit messages to RabbitMQ to trigger task execution, and eventually retrieve the result at a later time

6. Create a directory downloaderApp to hold our new python module, and a directory downloadedFiles where the downloaded files will be stored:

7. Create a downloaderApp.py module that will contain two functions, download and list, that will be the asynchronous tasks. 
  The following line is critical:

app = Celery('downloaderApp', backend='rpc://', broker='pyamqp://guest@localhost//')

This line creates:


A Celery application named downloaderApp


A broker on the localhost that will accept message via *Advanced Message Queuing Protocol (AMQP), the protocol used by RabbitMQ


A response backend where workers will store the return value of the task so that clients can retrieve it later (remember that task execution is asynchronous). If you omit backend, the task will still run, but the return value will be lost. rpc means the response will be sent to a RabbitMQ queue in a Remote Procedure Call pattern.

data pipeline, tutorial 2

Tutorial: Building An Analytics Data Pipeline In Python

https://www.dataquest.io/blog/data-pipelines-tutorial/

data pipeline tutorial

How to Set Up a Task Queue with Celery and RabbitMQ

https://www.linode.com/docs/development/python/task-queue-celery-rabbitmq/

spacy training process

https://course.spacy.io/chapter4


# Start with blank English model nlp = spacy.blank('en') # Create blank entity recognizer and add it to the pipeline ner = nlp.create_pipe('ner') nlp.add_pipe(ner) # Add a new label ner.add_label('GADGET') # Start the training nlp.begin_training() # Train for 10 iterations for itn in range(10): random.shuffle(examples) # Divide examples into batches for batch in spacy.util.minibatch(examples, size=2): texts = [text for text, annotation in batch] annotations = [annotation for text, annotation in batch] # Update the model nlp.update(texts, annotations)

why import hash (imphash)

https://www.fireeye.com/blog/threat-research/2014/01/tracking-malware-import-hashing.html

One unique way that Mandiant tracks specific threat groups' backdoors is to track portable executable (PE) imports. Imports are the functions that a piece of software (in this case, the backdoor) calls from other files (typically various DLLs that provide functionality to the Windows operating system). To track these imports, Mandiant creates a hash based on library/API names and their specific order within the executable. We refer to this convention as an "imphash" (for "import hash"). Because of the way a PE's import table is generated (and therefore how its imphash is calculated), we can use the imphash value to identify related malware samples. We can also use it to search for new, similar samples that the same threat group may have created and used.

open source threat or vulnerability materails

https://otx.alienvault.com/

https://nvd.nist.gov/vuln/data-feeds

https://www.exploit-db.com/