Intro to Natural Language Processing

What Object Recognition does for images and video, Natural Language Processing (NLP) does for text. NLP extracts structured data from unstructured text. This means that NLP algorithms can extract meaning from a free-form email, a report, or text passed to it from a Speech Recognition algorithm.

How does this work? NLP algorithms are trained using a supervised learning approach. They are fed large amounts of categorized text, which they can use to determine details about new text they are provided. With unstructured text like a free-form email, NLP can be used to categorize the text by matching patterns of words with its training set. Like Speech Recognition, NLP is faced with a domain problem. If an algorithm has to parse medical record documents, its training set can be limited to medical documents. However, if an NLP algorithm must be able to parse many different types of documents, its training set must be significantly larger. In this case context becomes incredibly important. When a user types the word “produce” are they talking about creating something or are they talking about lettuce? When they type “moped”, are they talking about acting sad or about their motorized scooter? NLP algorithms need to be able to identify the context of passages in order to correctly discern the meaning of text.

NLP has several applications that can be useful for business. The first is categorizing or tagging text. If you give an NLP algorithm a news article, it can identify whether the article is about politics, business, sports, etc. It wouldn’t accomplish this using keywords. Instead using its training data set, it would construct a holistic view of what an article about politics for example would look like. Any new article evaluated by the system that is similar to the articles in its training set tagged as being about politics will also be categorized as a political article.

NLP can also work at a more granular level to extracting named entities. Instead of categorizing an entire article, it can identify people, places, things, quantities, and dates in a passage of text. “Victor, who works for Foundation AI in Los Angeles, wrote this article in 2018.” In this example, the named entities would be ‘Victor’, ‘Foundation AI’, ‘Los Angeles, ‘article’ and ‘2018’. Like Object Recognition, there are pre-trained Named Entity recognition algorithms that can carry out this task, but they need to be trained and tuned by experienced data scientists to make them accurate. The accuracy of these systems can be improved by training them on specific domains of information (legal, medical, etc.)

A combination of these two techniques can for example be used to scan and categorize an email sent to a company, tag the email to be forwarded to the right person in the organization, and extract the sender’s information to be entered into the company’s ticketing system, CMS, or CRM. Paired with a Robotic Process Automation (RPA) system, the email can be automatically forwarded and the sender’s information automatically input into the company’s database. We will discuss how AI can be used to expand the capabilities of RPA systems in another article.

Natural Language Processing deals with the challenge of understanding language. Once you’ve understood a request from a customer, you may wish to take an action using an RPA system like we’ve discussed earlier, or you may wish to either ask the customer for more information or provide the customer with information that might be useful to them. These actions require the use of Natural Language Generation, which we will detail in a future article.