How to set up XPath query for HTML parsing?

Question

Here is some HTML code from http://chem.sis.nlm.nih.gov/chemidplus/rn/75-07-0 in Google Chrome that I want to parse the website for some project.

<div id="names">
<h2>Names and Synonyms</h2>
<div class="ds"><button class="toggle1Col"title="Toggle display between 1 column of wider results and multiple columns.">&#8596;</button>
    <h3 id="yui_3_18_1_3_1434394159641_407">Name of Substance</h3>
    <ul>
        <li id="ds2">
        ``  <div>Acetaldehyde</div>
        </li>
    </ul>
</div>

I wrote a python script to help me do such a thing by grabbing the name under one of the sections, but it just isn't returning the name. I think it's my xpath query, suggestions?

from lxml import html
import requests  
import csv 

names1 = []

page = requests.get('http://chem.sis.nlm.nih.gov/chemidplus/rn/75-07-0') 
tree = html.fromstring(page.text)

//This will grab the name data 

names = tree.xpath('//*[@id="yui_3_18_1_3_1434380225687_700"]')

//Print the name data 
print 'Names: ', names 

//Convert the data into a string  
names1.append(names)

//Print the bit length 

print len(names1)

//Write it to csv 

b = open('testchem.csv', 'wb')  
a = csv.writer(b)  
a.writerows(names1)
b.close()
print "The end"

Answer 1

It is important to inspect the string returned by page.text and not just rely on the page source as returned by your Chrome browser. Web sites can return different content depending on the User-Agent, and moreover, GUI browsers such as your Chrome browser may change the content by executing JavaScript while in contrast, requests.get does not.

If you write the contents to a file

import requests
page = requests.get('http://chem.sis.nlm.nih.gov/chemidplus/rn/75-07-0') 
with open('/tmp/test', 'wb') as f:
     f.write(page.text)

and use a text editor to search for "yui_3_18_1_3_1434380225687_700" you'll find that there is no tag with that attribute value.

If instead you search for Name of Substance you'll find

<div><br>Search for this InChIKey on the <a href="http://www.google.com/search?q=%22IKHGUXGNUITLKF-UHFFFAOYSA-N%22" target="new" rel="nofollow">Web</a></div></div><div id="names"><h2>Names and Synonyms</h2><div class="ds"><button class="toggle1Col" title="Toggle display between 1 column of wider results and multiple columns.">&#8596;</button><h3>Name of Substance</h3><ul>
<li id="ds2"><div>Acetaldehyde</div></li>

Therefore, instead you could use:

In [219]: tree.xpath('//*[text()="Name of Substance"]/..//div')[0].text_content()
Out[219]: 'Acetaldehyde'

---

How this XPath was found:

Starting from the <h3> tag:

In [215]: tree.xpath('//*[text()="Name of Substance"]')
Out[215]: [<Element h3 at 0x7f5a290e85d0>]

The <div> tag that we want is not a child but rather it is a subchild of the parent of <h3>. Therefore, go up to the parent:

In [216]: tree.xpath('//*[text()="Name of Substance"]/..')
Out[216]: [<Element div at 0x7f5a290f02b8>]

and then use //div to search for all <div>s inside the parent:

In [217]: tree.xpath('//*[text()="Name of Substance"]/..//div')
Out[217]: 
[<Element div at 0x7f5a290e88e8>,
 <Element div at 0x7f5a290e8940>,
 ...]

The first div is the one that we want:

In [218]: tree.xpath('//*[text()="Name of Substance"]/..//div')[0]
Out[218]: <Element div at 0x7f5a290e88e8>

and we can extract the text using the text_content method:

In [219]: tree.xpath('//*[text()="Name of Substance"]/..//div')[0].text_content()
Out[219]: 'Acetaldehyde'