how to fetch a page

This is the place for queries that don't fit in any of the other categories.

how to fetch a page

Postby ochichinyezaboombwa » Tue Jul 16, 2013 4:17 pm

Hi,
I am having trouble getting content from a page, e.g. http://rapgenius.com/artists/2pac. If I urllib.urlopen.read() it, only a small portion is delivered. It is designed so that I have to scroll it (using mouse / kb) to get to the bottom of it.

There is javaScript at the bottom:
Code: Select all
<script type="text/javascript">if (!NREUMQ.f) { NREUMQ.f=function() {
NREUMQ.push(["load",new Date().getTime()]);
...
looks like I need to somehow run it ... (until it doesn't run anymore?), using webbrowser and/or v8 / phantomjs / ... ? I did some research but am stuck.

Any idea? I would totally appreciate any help.
ochichinyezaboombwa
 
Posts: 200
Joined: Tue Jun 04, 2013 7:53 pm

Re: how to fetch a page

Postby verb » Tue Jul 16, 2013 4:39 pm

http://rapgenius.com/songs?for_artist_p ... t%5D=title

why don't you try to fetch the pages one by one by passing the page parameter to 3 ,4 ,5 etc..
verb
 
Posts: 12
Joined: Fri Feb 22, 2013 8:15 pm

Re: how to fetch a page

Postby ochichinyezaboombwa » Tue Jul 16, 2013 4:53 pm

I very much appreciate your reply; however it raises yet more questions.

  • where did you get this url?
  • what are the parameters?
  • how it is related to the one I reported a problem with?
Thanks!
ochichinyezaboombwa
 
Posts: 200
Joined: Tue Jun 04, 2013 7:53 pm


Return to General Coding Help

Who is online

Users browsing this forum: Baidu [Spider] and 3 guests