Member-only story
NLP: RNNs to Transformers
I have been more conceptual than historical. Also, I have eschewed mathematical rigor to preserve readability.
(1) NLP BEFORE RNN
Before the advancement of Recurrent Neural Networks (RNNs) in Natural Language Processing (NLP), several techniques were used for processing and analyzing natural language data.
(1.1) Rule-Based Methods: In this method, a set of rules was defined by linguistic experts to analyze and process natural language data. The rules were based on the grammar, syntax, and semantics of the language. These rules were used to identify and extract the relevant information from the text.
For example, consider the following rule: If a sentence contains a subject, a verb, and an object, then it is a declarative sentence.
This rule can be used to identify declarative sentences in a text.
(1.2) Statistical Methods: Statistical methods were used to analyze large amounts of natural language data. These methods involved the use of probabilistic models and machine learning algorithms to identify patterns in the data.
For example, the Naive Bayes algorithm was used for text classification tasks, such as spam filtering.
