spacy training process

https://course.spacy.io/chapter4


# Start with blank English model nlp = spacy.blank('en') # Create blank entity recognizer and add it to the pipeline ner = nlp.create_pipe('ner') nlp.add_pipe(ner) # Add a new label ner.add_label('GADGET') # Start the training nlp.begin_training() # Train for 10 iterations for itn in range(10): random.shuffle(examples) # Divide examples into batches for batch in spacy.util.minibatch(examples, size=2): texts = [text for text, annotation in batch] annotations = [annotation for text, annotation in batch] # Update the model nlp.update(texts, annotations)

why import hash (imphash)

https://www.fireeye.com/blog/threat-research/2014/01/tracking-malware-import-hashing.html

One unique way that Mandiant tracks specific threat groups' backdoors is to track portable executable (PE) imports. Imports are the functions that a piece of software (in this case, the backdoor) calls from other files (typically various DLLs that provide functionality to the Windows operating system). To track these imports, Mandiant creates a hash based on library/API names and their specific order within the executable. We refer to this convention as an "imphash" (for "import hash"). Because of the way a PE's import table is generated (and therefore how its imphash is calculated), we can use the imphash value to identify related malware samples. We can also use it to search for new, similar samples that the same threat group may have created and used.

open source threat or vulnerability materails

https://otx.alienvault.com/

https://nvd.nist.gov/vuln/data-feeds

https://www.exploit-db.com/

a reinforcement learning example

Five basic elements about reinforcement learning: agent, state, environment, policy, reward.

Here is an example https://builtin.com/data-science/reinforcement-learning-python

Let's assume we are trying to train a cat. Here are something copied from the above article.

• The cat will be the “agent” that is exposed to the “environment.”
• The environment is a house/play-area depending on what you're teaching.
• The situation encountered is called the “state,” which is analogous for example, to your cat crawling under the bed or running. These can be interpreted as states.
• The agents react by performing actions to change from one “state” to another.
• After the change in states, we give the agent either a “reward” or a “penalty” depending on the action that is performed.
• The “policy” is the strategy of choosing an action for finding better outcomes.

1. States: The state is a complete description of the world. No piece of information present in the world is hidden. It can be a position, a constant or a dynamic. We mostly record these states in arrays, matrices or higher order tensors.
2. Action: Action is usually based on the environment, different environments lead to different actions based on the agent. Set of valid actions for an agent are recorded in a space called an action space. These are usually finite in number.
3. Environment: This is the place where the agent lives and interacts. For different types of environments, we use different rewards, policies, etc.
4. Reward and return: The reward function R is the one which must be tracked all the time in reinforcement learning. It plays a vital role in tuning, optimizing the algorithm and stop training the algorithm. It depends on the current state of the world, the action just taken, and the next state of the world.
5. Policies: Policy is a rule used by an agent for choosing the next action. These are also called the agent's brains.

problem with ElasticSearch connection error

Following https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-18-04, while testing the elasticsearch as suggested,


curl -x GET "localhost:9200"

I received an error as following:

Error:

curl (7): Failed to connect to localhost port 9200: Connection refused

So what can I do? 

Using sudo service elasticsearch status, there is a red light. It tells me there was not insufficient memory for the Java Virtual Machine.

Then modify /etc/elasticsearch/jvm.options file:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

#-Xms2g
#-Xms2g

-Xms512m
-Xmx512m

run sudo systemctrl restart elasticsearch again,

check the status: sudo service elasticsearch status. You will see the green light.

Mac OS Log Analysis (2)

• System Log Folder: /var/log
• System Log: /var/log/system.log
• Mac Analytics Data: /var/log/DiagnosticMessages
• System Application Logs: /Library/Logs
• System Reports: /Library/Logs/DiagnosticReports
• User Application Logs: ~/Library/Logs (in other words, /Users/NAME/Library/Logs)
• User Reports: ~/Library/Logs/DiagnosticReports (in other words, /Users/NAME/Library/Logs/DiagnosticReports)

Mac OS Log Analysis (1)

Recently, my Mac laptop's (Majove, MacBook Pro (15-inch, 2017)) Safari failed to open. I searched around and could not find a solution. Even our company's helpdesk could not find a solution. Then I am curious to examine the log and try to figure it why the Safari launch failure happened.

Right now, the Safari still does not work. I solely use Chrome for web browsing.

Meanwhile, I am doing some research on cyber security. There are many works on examining on Windows or Linux logs and not so many focuses on the MacOS.

I examined the OSX Collecto (https://github.com/Yelp/osxcollector/blob/master/osxcollector/osxcollector.py) and feel that is not what I want. I begin to write some tools to analyze MacOS logs.

There is another post on this topic too. http://macadmins.psu.edu/wp-content/uploads/sites/24696/2016/06/psumac2016-19-osxlogs_macadmins_2016.pdf

Here I am summarizing those analysis step by step. I wish those will help our readers.