So I'm web scraping Google and am pretty sure it's blocking my requests based on the IP address. I've deployed my app to Heroku (which has dynamic IP addresses when the dynos restart) and I've noticed that if the app is up, after 5 requests, they stop being able to scrape properly. If I restart the dyne, then I get another 5 requests before it stops being able to scrape. This leads me to believe the static IP address when the dyno is up is the issue. I looked into QuotaGuard Dynamic IP's (https://devcenter.heroku.com/articles/quotaguard) but I don't think that will work because Google seems to be https. Has anyone dynamically proxied their requests through different IPs on Heroku before (if so, what do you recommend using)? I'm working in a Node.js environment.
How to change the IP address on each Heroku request?
1.1k Views Asked by nickcoding2 At
1
There are 1 best solutions below
Related Questions in NODE.JS
- How to solve CERT_UNTRUSTED error in nodemailer
- Run a loop over a callback, node js
- Implementing prerender.io middleware in sails.js
- Token based authorization in nodejs/ExpressJs and Angular(Single Page Application)
- formatting path string in javascript
- One to One screensharing using WEBRTC
- Create polygon from grid (for collisions)
- Strange npm behavior when installing packages like grunt
- Convert JSON.gz to JSON in node js
- "Your npm version is outdated." but it's not. While install yo
- Why put methods on the prototype of a class instead of declaring them in the constructor?
- Node JS Async Response
- mongoose get property from nested schema after `group`
- Cannot Receive Incoming call on Twilio android Client
- How can I change a specific line in a file with node js?
Related Questions in HEROKU
- NoMethodError: undefined method `update_average_rating' for nil:NilClass
- Using Amazon KMS service on Heroku
- How to have Heroku build my development branch on a staging server?
- Heroku not fetching from DB when scheduler running
- Search for a key in django.core.cache
- Ruby app fails in https request on heroku.com
- Rails: NameError: uninitialized constant Rails::Application::YAML
- clojure worker-only app on heroku fails with Error R10
- Clockwork not logging to Heroku console
- Javascript working in development locally but not production on Heroku
- Heroku PostgreSQL Studio not finding databases
- Heroku : Username for 'https://git.heroku.com': git
- Pg backups curl latest dump from Heroku
- How do you deploy build artifacts to Heroku from Codeship?
- Custom validation message (in model file) that includes a path not working
Related Questions in WEB-SCRAPING
- Scraping location data in rvest
- Python Beautiful Soup Table Data Scraping Specific TD Tags
- VBA: Extract HTML from new page (same url)
- Nokogiri how to traverse every row of a table with two classes
- URL Variable is not being recognized using NSURL
- Scrapy CrawlSpider not following links
- Scraping blog and saving date to database causes DateError: unknown date format
- Can Nokogiri interpret javascript? - Web Scraping
- Beautifulsoup: Getting a new line when I tried to access the soup.head.next_sibling value with Beautifulsoup4
- Web scraping with python and selenium
- getting specific images from page
- Why does Selenium return the source of the previously loaded page in Python?
- R 3.1.3 How to Scrape Multiple City-Data.com Records?
- How to eliminate certain elements when scraping?
- Parse an HTML table with Nokogiri in Ruby
Related Questions in DYNO
- Heroku Procfile not seeing my web: definition
- "App crashed" Django and Node together on Heroku
- Setting custom process as Heroku web process type
- Can you have Hobby and Standard Dynos at the same time in Heroku?
- Trouble deploying Nodejs App + Python on Heroku
- How to create permanent files on Heroku?
- Heroku Dyno inexplicably down?
- Scheduling tasks to run on heroku - How does it work in detail?
- how do i keep myself logged in heroku (chromedriver) even after dyno restarts everyday?
- Dyno Hours for paid account and Frequency to call app endpoint
- What does Herokus metric graph display?
- Can I set Heroku to not retry requests on timeout by default?
- Can I have one heroku application with multiple web dynos?
- Heroku: stop all one-off dynos without dyno name
- Why do I lose contents of my database after a heroku dyno restart?
Related Questions in QUOTAGUARD
- How to setup FTP static ip with QuotaGuard static?
- LD_PRELOAD not working on Heroku + jemalloc + quotaguard
- How to change the IP address on each Heroku request?
- Connect Heroku app to Atlas MongoDB using static IP
- Can't connect SFTP(AWS EC2) with QuotaGuard Static IP
- Proxy Heroku App Running Next.js and Prisma Client
- Where in python requests is curl --proxy-insecure analogue?
- Risk of Using 0.0.0.0/0 for MongoDB
- Regularly get Got an error reading communication packets populating mysql from python3 using quota guard. Any advice?
- Whitelist of Heroku Static IP QuotaGuard is timing out to connection to Aurora Postgres
- how to work locally with laravel when third-party api need only production IP to whitelist?
- Download a file from FTP via proxy with Ruby
- How to set angular proxy using QuotaGuard using node.js server
- How to set up QuotaGuard qgtunnel on pythonanywhere
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Use proxies, there are tons of paid / free proxies. You can rotate the proxies on each requests. Do note that there are different types of proxies. DataCenter IPs, Residential IPs and the most expensive Mobile IPs.