Introduction to NLP Deep Learning theories

Open books

In this post, I will summarize what I learnt from Natural Language Processing with Deep Learning offered by Stanford University, including the Winter 2017 video lectures, and the Winter 2019 lecture series. Both lectures were taught by Prof. Christopher Manning at Stanford University. Several topics on deep learning NLP theories which are covered in the two lectures include…

Read More

Who benefits from Sentiment Analysis?

Sentiment Analysis

With the evolution of technology and the growth of social media platforms, the interactions between businesses and customers are rapidly evolving. This has led to the generation of vast volumes of unstructured data. Every industry, company or organisation collects such data and requires that it is transformed into valuable, actionable insights that can be applied for business growth.

Read More

The next big move: Industries that can benefit from Text Analytics

Text analytics

Text Analytics is a machine learning technique that can interpret huge amounts of unstructured data which presents itself in the form of survey responses, feedback, chats, reviews, comments, emails, documents, records, data, and much more. Text Analytics or text mining can discover insights, trends, and structure. Tools like Sentiment Analysis, Emotion Detection, Topic Modeling, Named Entity Recognition, Offensive Language Detection, Tagging, Intent Analysis can help companies to understand the story behind the numbers and make informed decisions.

Read More

Recent advances in Transfer Learning for Natural Language Processing

An Overview of the Evolution of Recent Transfer Learning Techniques for Natural Language Processing (NLP). The following is an extract from my newly released book “Transfer Learning for Natural Language Processing”. The extract summarizes some recent NLP model architectures relying on the concept of transfer learning.

Read More

The evolution of Natural Language Processing and its impact on the legal sector

Natural language

Language is not just a set of independent meaningful words (a vocabulary), these words only take on meaning when they are understood in context. So, instead of trying to specify language with formal grammar rules and structures, you’re better off training the model (or your brain) with real-life examples. This is particularly true when NLP is being used in a legal setting, for example, for contract review and analysis.

Read More

Natural Language Processing Pipeline

NLP pipeline

A map for your studies and projectsBruno RodriguesJust now·6 min readIntroductionIn this article, I cover the development, or rather, a pipeline of an Natural Language Processing (NLP). Thus, interested parties can benefit from a guiding content in the development stages of the study or project.For those looking for a definition, NLP is a sub-area of ​​machine learning that works with natural language, whether dealing with text or audio. In technical terms, it studies the capabilities and limitations of machines to understand human language. A pratical application now is that while the text is written in English language, you probably are reading it in your native language, if not English. Is it truth? I also emphasize that Machine Learning (AM) is a sub-area of ​​Artificial Intelligence (AI).As shown in the image above, NLP is the result of the interdisciplinarity between Linguistics and ML. Goind deeper, NLP even uses Deep Learning (DL).There are two interesting concepts to mention: Natural Language Generation Systems (NLGS’s) and Natural Language Understanding Systems (NLUS’s). As far as NLG’s are concerned, they convert information from computer databases into human-understanble language and, as compared to the NLUS’s, theses convert human language occurances into more formal representations more easily manipulated by computer programs.Human language is not scientifically simple for a machine. One of its main challenging features is semantic ambiguity. We as humans have two fundamental factors that separate us from machines in terms of natural language: common cultural knowledge and prior experience.Other important factors are the context and tone of voice. Here you are faced with issues about feelings and emotions together. Machines do not contain feelings and emotions, nor can intrinsically and truly understand them.That said, it is possible to realize that NLP is a brige between human and machine. Furthermore, it is a great tool to help human beings in processes and automations of tasks that would streamline certain human needs, such as, for example, producing specific and everyday documents.Some other applications are sentiment analysis, contract review, machine translation, among others.Now, let’s start!PipelineWe can divide the NLP development process into five stages: i) data acquisition, ii) data cleaning, iii) pre-processing, iv) training, and v) data evaluation.Let’s start with data acquisition.Data acquisitionIn technical terms, data acquistion, when referring to NLP, is denominated corpus acquisition, for dataset in NLP is that way named. Some methods of acquiring data are web scraping and crawling. Selenium, Requests and Beautiful Soup are tools for this. It is possible to have and use data from databases such as SQL and Sparks. There is also the possibility of using structured corpus by third parties, such as COLT, IMDB reviws and Standford Sentiment Treebank. From theses, you develop a NLP initial model to put in interectaion with users or clients, to, thus, improve the model.You must be logical when acquiring a corpus, for the data must assist the needs of your project or study. Imagine two models, one aimed at education and the other at identifying toxicity. While you must eliminate offensive words for a model, for the other, you allow them. Another factor to be clean up is stereotyping trends.Data cleaningStopWords RemovalGenerally, prepositions and articles are removed. The idea is that words connecting ideas are removed. However, for other models, they are important, such as Translation, Question and Answer (Q&A) and Natural Language Understanding (NLU) tasks, wich can suffer from the loss of these words. Again, you must be logical in your work. Differently from these, a model that fits well this procedure is a spam detector.Below is an example of this procedure:Offensive words fit into this example, as stated above.Pre-processingBag-of-Words (BOW)This is simply the count of terms present in text. The procedure is you create a dictionary or list referring to the words present in your text and count their repetitions. Below is an illustration:Note that there are five sentences on the left side of the frame: i)It is a puppy, ii)It is a kitten, iii)It is a cat, iv)That is a dog and this is a pen and v)It is a matrix. Above the frame, there are words encounted from some of the sentences. Below them there is the presence count of those words, metioned above the frame, within sentences. If you add up all columns, you will receive the number of times that each word appear in the texts. Above we have structured data and unstructured data. From the unstructured data, the raw material, we build the structured data, the middle and fundamental part of the NLP process. It is necessary to pay attention for this method. It is not an absolute rule, it is necessary to master the method to know when to apply it and how to apply it.Through this method you need to be aware of the Curse of the Dimensionality so that your model is not inefficient. Another issue is the need to normalize the importance of words. Finally, I highlight that this method does not differentiate common words from more specific words. Imagine that a text has 100 words “very” and only one “basketball”. Although “very” appears 100 times in the text, nothing can be inferred about its content. However, just reading the word “basketball” once, it is possible to infer that the text talks about this sport. To solve this problem, the TF-IDF (Term Frequency-Inverse Document Frequency) was developed.TF-IDFThe idea behind this method is to first count an “a” term within an “x” document and divide it by the number of terms within the “x” document. It considers repetition of terms. After, there is the division between the number of documents by the number of documents, in wich “a” appears, being the quotient treated by log. Below is the mathematical formula:But, this model deals not with Curse of Dimensionality, as its approach considers the entire document to the point where many representations are equal to zero, not being significantly different from the Bag-of-Words. To solve this issue, the Word2Vec algorithm.Word2VecIt is an neural networks algorithm that does not represent an entire document. Its approach is to consider that the meaning of the words is given by the context, or rather by the neighboring words. I will not go into details about this algoritm, but I would like to point out that, for the subject of this topic, it has been the main tool.Finally I provide this explanatory schema, got on Google, of its objetive function:While the window scrolls through the text, the probability calculation for each word is given by the following formula:TrainingThere are numerous possibilities to apply NLP. To decide wich model to use, if in doubt, start by consulting the literature. Make a baseline. Use NLTK, Gensim and Spacy. It is not advisable to define something here when dealing with such comprehensive content.EvaluationJust as there are many tasks, there are many metrics. So, just like what was said above, consult the literature. There is already a considerable amount of work done demonstrating the performance of models and evaluation methods.Final wordsTo have a successful career, in terms of quality, it takes commitment, planning, study and action. I hope we can develop the world that way.Make your observations, corrections, tips below. I’m waiting for more learning.Grateful for your reading!

Read More

Top 10 Natural Language Processing (NLP) Tools for beginners

Glass window

Natural Language Processing (NLP) is one of the most interesting and fast-growing branches of Artificial Intelligence to work with. This realm is developing rapidly. Every year or even month there are new advancements. New tools are appearing, and existing ones are being updated with more progressive features. Having some experience in this field, I decided to share my best tools for NLP. My goal is not to provide you with dry analysis, but to advise you of the instruments that I enjoyed myself. Here, I gathered well-suited stuff for beginners. All links with documentations and guides are added.

Read More

[Paper Xplained] Neural Machine Translation using Bahdanau Attention

The paper in discussion is “Neural Machine Translation by Jointly Learning to Align and Translate” by Dzmitry Bahdanau, KyungHyun Cho & Yoshua Bengio. This is the paper that has introduced the now-famous “Attention Mechanism” in the year 2014. Though there are several advancements that happened to the concept of Attention, the mechanism introduced by this paper is still known as “Bahdanau Attention” or “Additive Attention”

Read More