24

Python. Scrapy Xpath returns an empty array

 4 years ago
source link: https://www.codesd.com/item/python-scrapy-xpath-returns-an-empty-array.html
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

Python. Scrapy Xpath returns an empty array

advertisements

I am using scrapy to scrape information from a website in python and I am only getting used to using Xpaths to find information.

I want to return a list of all the average ratings from albums of this artist from this page. https://rateyourmusic.com/artist/kanye_west

To find the node for the albums I used //div[@id="disco_type_s"] and I tried searching children for divs with the attribute disco_avg_rating using div[@class="disco_avg_rating"]/text()

Here is my function

def parse_dir_contents(self, response):
    item = rateyourmusicalbums() *ignore this

    for i in response.xpath('//div[@id="disco_type_s"]'):
        item['average rating']=i.xpath('div[@class="disco_avg_rating"]/text()').extract()
        yield item

Everything I try to get this list causes a problems. Usually it's more straight but this time I have to differentiate between albums and singles etc. so I am having troubles.

Appreciate your help, I am fairly new to web scraping.

response.xpath('//div[@id="disco_type_s"]') finds only one tag (this is what mostly happens when using id to match the xpath, they are unique). to get a list of selectors you should use something like:

response.xpath('//div[@id="disco_type_s"]/div[@class="disco_release"]') which will match multiple tags, so you can iterate on those.

then get the average rating with './div[@class="disco_avg_rating"]/text()'


report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK