celery and rabbitmq data pipeline example on mac OS

From  https://www.linode.com/docs/development/python/task-queue-celery-rabbitmq/

Install celery and rabbitmq
1) pip install celery
2) brew update
3) brew install rabbitmq
4) export PATH=$PATH:/usr/local/opt/rabbitmq/sbin

5) rabbitmq-server

   Then http://localhost:15672/ will give you access.

5. Celery application has two parts: 

• Workers that wait for messages from RabbitMQ and execute the tasks.
• Client that submit messages to RabbitMQ to trigger task execution, and eventually retrieve the result at a later time

6. Create a directory downloaderApp to hold our new python module, and a directory downloadedFiles where the downloaded files will be stored:

7. Create a downloaderApp.py module that will contain two functions, download and list, that will be the asynchronous tasks. 
  The following line is critical:

app = Celery('downloaderApp', backend='rpc://', broker='pyamqp://guest@localhost//')

This line creates:


A Celery application named downloaderApp


A broker on the localhost that will accept message via *Advanced Message Queuing Protocol (AMQP), the protocol used by RabbitMQ


A response backend where workers will store the return value of the task so that clients can retrieve it later (remember that task execution is asynchronous). If you omit backend, the task will still run, but the return value will be lost. rpc means the response will be sent to a RabbitMQ queue in a Remote Procedure Call pattern.

data pipeline, tutorial 2

Tutorial: Building An Analytics Data Pipeline In Python

https://www.dataquest.io/blog/data-pipelines-tutorial/

data pipeline tutorial

How to Set Up a Task Queue with Celery and RabbitMQ

https://www.linode.com/docs/development/python/task-queue-celery-rabbitmq/

spacy training process

https://course.spacy.io/chapter4


# Start with blank English model nlp = spacy.blank('en') # Create blank entity recognizer and add it to the pipeline ner = nlp.create_pipe('ner') nlp.add_pipe(ner) # Add a new label ner.add_label('GADGET') # Start the training nlp.begin_training() # Train for 10 iterations for itn in range(10): random.shuffle(examples) # Divide examples into batches for batch in spacy.util.minibatch(examples, size=2): texts = [text for text, annotation in batch] annotations = [annotation for text, annotation in batch] # Update the model nlp.update(texts, annotations)

why import hash (imphash)

https://www.fireeye.com/blog/threat-research/2014/01/tracking-malware-import-hashing.html

One unique way that Mandiant tracks specific threat groups' backdoors is to track portable executable (PE) imports. Imports are the functions that a piece of software (in this case, the backdoor) calls from other files (typically various DLLs that provide functionality to the Windows operating system). To track these imports, Mandiant creates a hash based on library/API names and their specific order within the executable. We refer to this convention as an "imphash" (for "import hash"). Because of the way a PE's import table is generated (and therefore how its imphash is calculated), we can use the imphash value to identify related malware samples. We can also use it to search for new, similar samples that the same threat group may have created and used.

open source threat or vulnerability materails

https://otx.alienvault.com/

https://nvd.nist.gov/vuln/data-feeds

https://www.exploit-db.com/

a reinforcement learning example

Five basic elements about reinforcement learning: agent, state, environment, policy, reward.

Here is an example https://builtin.com/data-science/reinforcement-learning-python

Let's assume we are trying to train a cat. Here are something copied from the above article.

• The cat will be the “agent” that is exposed to the “environment.”
• The environment is a house/play-area depending on what you're teaching.
• The situation encountered is called the “state,” which is analogous for example, to your cat crawling under the bed or running. These can be interpreted as states.
• The agents react by performing actions to change from one “state” to another.
• After the change in states, we give the agent either a “reward” or a “penalty” depending on the action that is performed.
• The “policy” is the strategy of choosing an action for finding better outcomes.

1. States: The state is a complete description of the world. No piece of information present in the world is hidden. It can be a position, a constant or a dynamic. We mostly record these states in arrays, matrices or higher order tensors.
2. Action: Action is usually based on the environment, different environments lead to different actions based on the agent. Set of valid actions for an agent are recorded in a space called an action space. These are usually finite in number.
3. Environment: This is the place where the agent lives and interacts. For different types of environments, we use different rewards, policies, etc.
4. Reward and return: The reward function R is the one which must be tracked all the time in reinforcement learning. It plays a vital role in tuning, optimizing the algorithm and stop training the algorithm. It depends on the current state of the world, the action just taken, and the next state of the world.
5. Policies: Policy is a rule used by an agent for choosing the next action. These are also called the agent's brains.