a reinforcement learning example

Five basic elements about reinforcement learning: agent, state, environment, policy, reward.

Here is an example https://builtin.com/data-science/reinforcement-learning-python

Let's assume we are trying to train a cat. Here are something copied from the above article.

• The cat will be the “agent” that is exposed to the “environment.”
• The environment is a house/play-area depending on what you're teaching.
• The situation encountered is called the “state,” which is analogous for example, to your cat crawling under the bed or running. These can be interpreted as states.
• The agents react by performing actions to change from one “state” to another.
• After the change in states, we give the agent either a “reward” or a “penalty” depending on the action that is performed.
• The “policy” is the strategy of choosing an action for finding better outcomes.

1. States: The state is a complete description of the world. No piece of information present in the world is hidden. It can be a position, a constant or a dynamic. We mostly record these states in arrays, matrices or higher order tensors.
2. Action: Action is usually based on the environment, different environments lead to different actions based on the agent. Set of valid actions for an agent are recorded in a space called an action space. These are usually finite in number.
3. Environment: This is the place where the agent lives and interacts. For different types of environments, we use different rewards, policies, etc.
4. Reward and return: The reward function R is the one which must be tracked all the time in reinforcement learning. It plays a vital role in tuning, optimizing the algorithm and stop training the algorithm. It depends on the current state of the world, the action just taken, and the next state of the world.
5. Policies: Policy is a rule used by an agent for choosing the next action. These are also called the agent's brains.

problem with ElasticSearch connection error

Following https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-18-04, while testing the elasticsearch as suggested,


curl -x GET "localhost:9200"

I received an error as following:

Error:

curl (7): Failed to connect to localhost port 9200: Connection refused

So what can I do? 

Using sudo service elasticsearch status, there is a red light. It tells me there was not insufficient memory for the Java Virtual Machine.

Then modify /etc/elasticsearch/jvm.options file:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

#-Xms2g
#-Xms2g

-Xms512m
-Xmx512m

run sudo systemctrl restart elasticsearch again,

check the status: sudo service elasticsearch status. You will see the green light.

Mac OS Log Analysis (2)

• System Log Folder: /var/log
• System Log: /var/log/system.log
• Mac Analytics Data: /var/log/DiagnosticMessages
• System Application Logs: /Library/Logs
• System Reports: /Library/Logs/DiagnosticReports
• User Application Logs: ~/Library/Logs (in other words, /Users/NAME/Library/Logs)
• User Reports: ~/Library/Logs/DiagnosticReports (in other words, /Users/NAME/Library/Logs/DiagnosticReports)

Mac OS Log Analysis (1)

Recently, my Mac laptop's (Majove, MacBook Pro (15-inch, 2017)) Safari failed to open. I searched around and could not find a solution. Even our company's helpdesk could not find a solution. Then I am curious to examine the log and try to figure it why the Safari launch failure happened.

Right now, the Safari still does not work. I solely use Chrome for web browsing.

Meanwhile, I am doing some research on cyber security. There are many works on examining on Windows or Linux logs and not so many focuses on the MacOS.

I examined the OSX Collecto (https://github.com/Yelp/osxcollector/blob/master/osxcollector/osxcollector.py) and feel that is not what I want. I begin to write some tools to analyze MacOS logs.

There is another post on this topic too. http://macadmins.psu.edu/wp-content/uploads/sites/24696/2016/06/psumac2016-19-osxlogs_macadmins_2016.pdf

Here I am summarizing those analysis step by step. I wish those will help our readers.

Deep Dive: Examining A PowerShell Payload

https://blog.huntresslabs.com/deep-dive-examining-a-powershell-payload-6b4325b0f1a4

adversarial machine learning (1)

This series are to help readers to get familiar with different adversarial attacks in the adversarial machine learning domain.

We are going through different approaches introduced in https://adversarial-robustness-toolbox.readthedocs.io/en/latest/index.html

interpreting black-box machine learning models

https://youtu.be/lpuvHJipm0s

How To Improve Deep Learning Performance

https://machinelearningmastery.com/improve-deep-learning-performance/

Role of a Data Science Manager

https://medium.com/sequoia-capital/role-of-a-data-science-manager-2c69264511d3

NLP topics (1)

There are many areas using NLP technique to understand texts and provide intelligence to customers.
Here are some:

1. word cloud
2. heat map
3. sentiment analysis
4. graph analysis
5. cluster analysis
6. classification
7. auto text generation
8. text caption for images
9. generating image from text

human in computer world

We have been talking about the reinforcement learning a lot, especially after Google's deep mind defeated human Go players.

In another angle, how to really describe a human in a computer world? How can the human grow from infant, to toddler, to youth and adult?

What are the environment? What are the reward/feedback?

If we could record a twin's growth path, mainly image, audio, sensory, etc, that will be cool to analyze, model and create a virtual human growth trajectory.

super charged neurons in AI

I am working in the Artificial Intelligence area. I am thinking of two events happened to me.

1) I have the same dream over many years. While writing the post, I could not recall what exactly the dream now.

2) While I drove to pass a large field of peach flowers, I recalled the moment that I biked to high school and passed a large field of peach flowers.

Both events normally would not come to my mind at all. It seems deep in the brain, there are some super charged neurons which store some unique information, can be trigged by some specific outside stimulus.

I believe in the AI algorithm design, we should add such neurons. It stays there for a specific event, occasionally trigged by a specific type of stimulus.

topic modeling with LSA, PLSA, LDA& LDA2VEC by Joyce Xu

https://www.topbots.com/nlu-topic-modeling/?utm_source=ActiveCampaign&utm_medium=email&utm_content=Practical+uses+of+topic+++language+models+in+NLP+applications&utm_campaign=Weekly+Newsletter+05+01+2019+Issue+139

top 10 papers in reinforcement learning in 2018 by Mariya Yao

https://www.topbots.com/most-important-ai-reinforcement-learning-research/?utm_source=ActiveCampaign&utm_medium=email&utm_content=Is+reinforcement+learning+finally+useful+for+the+real+world%3F&utm_campaign=Weekly+Newsletter+02+13+2019+Issue+128

three types of quantum computer

1. Analog quantum computer (quantum annealer, adiabatic QC, direct quantum simulation) examples: D-wave
2. Noisy intermediate-scale quantum (NISQ) gate-based computer  
1. example: IBM quantum computer
3. Fully error-corrected gate-based quantum computers. 
1. not available yet

Two leading technologies for quantum computing—trapped ions and superconducting qubits—use very different strategies for embodying and operating on qubits. 

quantum computer difficuty

There are many technical challenges:

1) quits cannot intrinsically reject noise

2) error free quantum computer requires quantum error correction

3) large data inputs cannot be loaded into a quantum computer efficiently.

4) Quantum Computers Will Need a New Software Stack

5) The Intermediate State of a Quantum Computer Cannot Be Measured Directly

From "Quantum Computing: Progress and Prospects (2018)" by National Academies Press. The pdf file is http://nap.edu/25196