[cs224n] Lecture 1 Natural Language Processing with Deep Learning

Natural Language Processing with Deep Learning

최근에 NLP 관련해서 stanford의 cs224n 강의를 듣게 됐다. 

강의 하나씩 들으면서 간단하게 정리도 하면서 나중에 다시 볼 겸 블로그에 남기려고 한다.

Plan

1. What is Natural Language Processing? The nature of human language
2. What is Deep Learning
3. Why is language understanding difficult
4. Intro to the application of Deep Learning of NLP

What is Natural Language Processing?

NLP Applications

• Spell checking, keyword search, finding synonyms
• Extracting information from websites
  • product price, dates, location, people or company names
• Classifying: reading level of school texts, positive/negative sentiment of longer documents
• Machine translation
• Spoken dialogue
• Complex question answering

What’s special about human language?
A human language is a system specifically constructed to convey the speaker/writer’s meaning

• Not just an environmental signal, it’s a deliberate communication
• Using an encoding which little kids can quickly learn

A human language is a discrete/symbolic/categorical signaling system

---

What’s Deep Learning

• Deep learning is a subfield of machine learning

• Most machine learning methods work well because of human-designed representations and input features

• Machine learning becomes just optimizing weights to best make a final prediction

• Representation learning attempts to automatically learn good features or representations

• Deep learning algorithms attempt to learn representation and an output

Raw signals -> representation that can be used for features

Reasons for Exploring Deep Learning

• Manually designed features are often over-specified, incomplete and take a long time to design and validate
• Learned Features are easy to adapt, fast to learn
• Deep learning provides a very flexible, universal, learnable framework for representing world, visual and linguistic information
• Deep learning can learn unsupervised and supervised

Other reasons

• Large amounts of training data favor deep learning
• Faster machines and multicore CPU/GPUs favor Deep learning
• New models, algorithms, ideas
  • Better ,more flexible learning of intermediate representations
  • Effective end-to-end joint system learning
  • Effective learning methods for using context and transferring between tasks

Improved performance

---

• Multivariate Calculus, Linear Algebra - MATH 51, CME 100
• Basic Probability and Statistics - CS 109
• Fundamentals of Machine Learning - CS229 or CS221
  • loss functions
  • taking simple derivatives
  • performing optimization with gradient descent

---

Why is NLP hard

왜 NLP 가 어려운지 이해를 해야한다…

• Complexity in representing, learning and using linguistic/situational/world/visual knowledge
• Human languages are ambiguous (unlike programming and other formal languages)
• Human language interpretation depends on real world, common sense, and contextual knowledge

---

People & Papers

• Danqi Chen
  • A Fast and Accurate Dependency Parser using Neural Networks / EMNLP 2014 / SyntaxNet(?)
  • Reading Wikipedia to Answer Open-Domain Questions / ACL 2017