Why is NLU not picking up an entire text of data

463 Views Asked by At

I appears that NLU is not recognizing the full blob of data I am providing it. Am I doing something wrong in my code or have misplaced assumptions on how the api should work? The response from the api is included and it contains the code that was analyzed as well as the full submitted text. There is a delta and I'm unsure why that is.

Here's my code:

def nlu(text):
print("Calling NLU")
url = "https://gateway.watsonplatform.net/natural-language-understanding/api/v1/analyze?version=2017-02-27"
data = {
    'text': text,
    'language': "en",
    'return_analyzed_text': True,
    'clean': True,
    'features': {
        'entities': {
            'emotion': True,
            'sentiment': True,
            'limit': 2
        },
        "concepts": {
            "limit": 15
        },
        'keywords': {
            'emotion': True,
            'sentiment': True,
            'limit': 2
        }
    }
}
headers = {
    'content-type': "application/json"
}
username = os.getenv("nlu-username")
password = os.getenv("nlu-password")
print("NLU", username, password)
print("data", json.dumps(data))
response = requests.request("POST", url, data=json.dumps(data), headers=headers, auth=(username, password))
print("Done calling NLU")
print(response.text)

Here's the request/response:

"keywords": [
{
  "text": "anthropologists study skeletons",
  "sentiment": {
    "score": 0.0
  },"analyzed_text": "move between two thousand eight and two thousand twelve archaeologists excavated the rubble of an ancient hospital in England in the process they uncovered a number of skeletons one in particular belong to a wealthy Mel who lived in the eleventh or twelfth century and died of leprosy between the ages of eighteen and twenty five how do we know all this simply by examining some old soil Kate bones even centuries after death skeletons carry unique features that tell us about their identities and using modern tools and techniques we can read those features as clues this is a branch of science known as biological anthropology it allows researchers to piece together details about Incheon individuals and identify historical events that affected whole populations when researchers uncover a skeleton some of the first clues they gather like age and gender line its morphology which is the structure appearance and size of a skeleton mostly the clavicle stop growing at age twenty five so a skeleton with the clavicle that hasn't fully formed must be younger than similarly the plates in the cranium can continue fusing up to age forty and sometimes beyond by combining these with some microscopic skeletal clues physical anthropologists can estimate an approximate age of death meanwhile pelvic bones reveal gender biologically female palaces are wider allowing women to give birth whereas males are narrower those also betrayed the signs of aging disease disorders like anemia leave their traces on the bones and the condition of teeth can reveal clues to factors like diet and malnutrition which sometimes correlate with wealth or poverty a protein called collagen can give us even more profound details the air we breathe water we drink and food we eat leaves permanent traces in our bones and teeth in the form of chemical compounds these compounds contain measurable quantities called isotopes stable isotopes in bone collagen and tooth enamel varies among mammals dependent on where they lived and what they eat so but analyzing these isotopes we can draw direct inferences regarding the diet and location of historic people not only that but during life bones undergo a constant cycle of remodeling so if someone moves from one place to another bones synthesized after that move will also reflect the new isotopic signatures of the surrounding environment that means that skeletons can be used like migratory maps for instance between one and six fifty A. D. the great city of TOT Makana Mexico bustled with thousands of people researchers examined the isotope ratios and skeletons to the now which held details of their diets when they were young they found evidence for significant migration into the city a majority of the individuals were born elsewhere with further geological and skeletal analysis they may be able to map where those people came from that work in tier two Akon is also an example of how bio anthropologists study skeletons in cemeteries and mass graves and analyze their similarities and differences from not information they can learn about cultural beliefs social norms wars and what caused their deaths today we use these tools to answer big questions about how forces like migration and disease shape the modern world DNA analysis is even possible in some relatively well preserved ancient remains that's helping us understand how diseases like tuberculosis have evolved over the centuries so we can build better treatments for people today ocean skeletons can tell us a surprisingly great deal about the past two of your remains are someday buried intact what might archaeologists of the distant future learn from them"
2

There are 2 best solutions below

3
On

I just tried on NLU with your text and getting a proper response. Check the below result. I think you should try the Watson API Explorer first with your service credentials. It will also help you to fix any misplaced headers or missed param into the API call.

Note: Just remove the "metadata": {} in the parameter object before making the POST call as it's for URL and HTML.

{
"semantic_roles": [{
    "subject": {
        "text": "anthropologists"
    },
    "sentence": "anthropologists study skeletons",
    "object": {
        "text": "skeletons"
    },
    "action": {
        "verb": {
            "text": "study",
            "tense": "present"
        },
        "text": "study",
        "normalized": "study"
    }
}],
"language": "en",
"keywords": [{
        "text": "anthropologists",
        "relevance": 0.966464
    },
    {
        "text": "skeletons",
        "relevance": 0.896147
    }
],
"entities": [],
"concepts": [{
    "text": "Cultural studies",
    "relevance": 0.86926,
    "dbpedia_resource": "http://dbpedia.org/resource/Cultural_studies"
}],
"categories": [{
        "score": 0.927751,
        "label": "/science/social science/anthropology"
    },
    {
        "score": 0.219365,
        "label": "/education/homework and study tips"
    },
    {
        "score": 0.128377,
        "label": "/science"
    }
],
"warnings": [
    "emotion: cannot locate keyphrase",
    "relations: Not Found",
    "sentiment: cannot locate keyphrase"
]

}

3
On

In your code you have

data=json.dumps(data)

Which is converting the whole JSON object to a string. That should just be:

data=data

Also I would recommend to use the Python WDC SDK, as it will make it easier for you.

The same example as above.

import json
from watson_developer_cloud import NaturalLanguageUnderstandingV1
import watson_developer_cloud.natural_language_understanding.features.v1 as Features

username = os.getenv("nlu-username")
password = os.getenv("nlu-password")

nluv1 = NaturalLanguageUnderstandingV1(
    username=username, 
    password=password)

features = [ 
    Features.Entities(), 
    Features.Concepts(),
    Features.Keywords()
]

def nlu(text):
    print('Calling NLU')
    response = nluv1.analyze(text,features=features, language='en')
    print('Done calling NLU')
    print(json.dumps(response, indent=2))