I want to crawl web page like quora, pinterest, etc. I found all examples using selenium to simulate scrolling action, yet open a new window for scrolling and crawling is slow and inefficient. Is Any method for crawling an infinite scrolling page more efficient?
Can any method crawl a infinite scrolling web in python without selenium?
1k Views Asked by Leo Hsieh At
1
There are 1 best solutions below
Related Questions in WEB-SCRAPING
- Scraping location data in rvest
- Python Beautiful Soup Table Data Scraping Specific TD Tags
- VBA: Extract HTML from new page (same url)
- Nokogiri how to traverse every row of a table with two classes
- URL Variable is not being recognized using NSURL
- Scrapy CrawlSpider not following links
- Scraping blog and saving date to database causes DateError: unknown date format
- Can Nokogiri interpret javascript? - Web Scraping
- Beautifulsoup: Getting a new line when I tried to access the soup.head.next_sibling value with Beautifulsoup4
- Web scraping with python and selenium
- getting specific images from page
- Why does Selenium return the source of the previously loaded page in Python?
- R 3.1.3 How to Scrape Multiple City-Data.com Records?
- How to eliminate certain elements when scraping?
- Parse an HTML table with Nokogiri in Ruby
Related Questions in WEB-CRAWLER
- Scrapy CrawlSpider not following links
- python scrapy login redirecting problems
- Google spider gives 404 error on Angular links: how to fix it?
- Watson Content Analytics: How to make web crawler plug-in to get data, sending POST request?
- scrapy startproject error
- Crawler architecture: Avoid getting requests counted in Google Analytics
- application.cfc - conditionally turn on session and/or client management?
- Sails.js static html renderer
- How to download text contained in JavaScript files via crawler4j?
- T_STRING error in my php code
- Select option from dropdown and submit request using nodejs
- Web-Crawler for VBA
- How to extract the content of an HTML attibute
- No performance gain with python threading
- Delay when extracting email
Related Questions in INFINITE-SCROLL
- Infinite looped vertical slider with no pause
- Angular select scroll event not fired
- QtQuick2 - QML - create infinite moving line animation
- Infinite scroll on single.php with URL change
- RecyclerView deleting item doesn't call OnScrollListener
- Tumblr Like Buttons are Unclickable with Infinite Scroll
- How can I scrape the correct number of URLs from an infinite-scroll webpage?
- Mvc-infinite-scroll package null session
- Meteor pagination: cursor fetch limit with getmore
- How to create multiple Load More buttons on a page (WordPress)
- call a function when user scrolls near bottom of page
- Endless Scrolling Background jumps at certain Viewport sizes
- Detecting when the bottom of a scrollable div has been reached with an offset
- ng-repeat limitTo on page scroll to bottom
- Infinite Scroll: show elements between a range
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
If the website uses Ajax frame to transfer data, my answer could help you. https://stackoverflow.com/a/34802775/5246180