Helping Machines Understand Complex Questions
source link: https://towardsdatascience.com/helping-machines-understand-complex-questions-48536fd93ef2?gi=4601ef609a6e
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
(This project was inspired by Google QUEST Kaggle Challenge )
The field of Natural Language Processing has always been fascinating to me. To imagine that a computer can understand and respond to human language is nothing short of amazing. I’ve been wanting to work on a project that involved both NLP and also deep learning for a little while, so I decided to create a project where a stakeholder needs to improve his Question & Answer system.
As advanced as QA systems and chatbots have become, they still struggle when faced with questions that are more complex in nature. Machines can easily compute dimensions such as number of characters, or vocabulary, but humans are much better at computing qualitative metrics such as “is this question looking for an opinion or recommendation?”
A model that can predict such subjective aspects of questions and correctly identify complexity can be very helpful in improving how these tools are built and implemented.
You can find the data we will be using here . The original data consists of over six thousand samples of question-answer pairs from various StackExchange and 30 human-generated subjective labels for each sample. I’ve adapted the problem from this competition to a more business-like situation. Our imaginary stakeholder is only interested in a machine-based solution to evaluate the questions it receives from its customers, and only over a few subjective aspects. The final goal is for his QA system to automatically answer the questions that are more factual and flag complex questions for employees to answer them.
So, we have a multi-class, multi-label text classification problem, where a sample can be classified for more than one class at the same time, and 6,079 samples to work with. We will use GloVe pre-trained word embeddings and a neural network model with Keras. Let’s start!
In order to not make this post too long, we will focus on the natural language preprocessing and modeling aspects, but you can find the entirety of this project including data exploration on my Github page .
Recommend
-
24
When humans and robots cross paths, the results aren’t just frustrating—the autonomous car, say, that’s too shy to turn left—they can also be fatal. Consider last year’s Uber crash, in which the self-driving algorithms
-
4
Complex traces you can actually understand by leveraging Activity Tracing January 1...
-
5
-
5
Helping RPM stay afloat on big fleets of machines I've been doing some work lately which touches a lot of Red Hat-derived machines. This is not a new pattern in my life -- they always tend to turn up in one place or another. I f...
-
5
This is a guest blogpost by Amy Holder from Neo4j. She examines recent interest in graph databases as the basis for the next wave of human capital management technology.Content Continues Below
-
4
My attempt to understand why people perceive Java as complexSkip to contentShare this:I work as an Archi...
-
2
Machines are needed to find complex software problems, humans can’t do it, says AI firm
-
1
Understand and retain complex text with the help of AIRanked #14 for todayWelmaUnderstand and retain complex text with the help of AI
-
2
TechAlibaba launches AI model that can understand images and have more complex conve...
-
4
How portable X-ray machines are helping remote patientsPublished1 day ago
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK