Bdtjtk

I've written a script in python in combination with selenium using proxies to get the text of differnt links populated upon navigating to a url, as in this one. What I want to parse from there is the visible text connected to each link.

The script I've tried so far with is capable of producing new proxies when this function start_script() is called within it. The problem is that the very url lead me to this redirected link. I can get rid off this redirection only when I keep trying on until the url is satisfied with a proxy. My current script can try twice only with two new proxies.

How can I use any loop within get_texts() function in such a way so that it will keep trying using new proxies until it parses the required content?

My attempt so far:

import requests

import random

from itertools import cycle

from bs4 import BeautifulSoup

from selenium import webdriver



link = 'http://www.google.com/search?q=python'



def get_proxies():   

    response = requests.get('https://www.us-proxy.org/')

    soup = BeautifulSoup(response.text,"lxml")

    proxies = [':'.join([item.select_one("td").text,item.select_one("td:nth-of-type(2)").text]) for item in soup.select("table.table tbody tr") if "yes" in item.text]

    return proxies



def start_script():

    proxies = get_proxies()

    random.shuffle(proxies)

    proxy = next(cycle(proxies))

    chrome_options = webdriver.ChromeOptions()

    chrome_options.add_argument(f'--proxy-server={proxy}')

    driver = webdriver.Chrome(chrome_options=chrome_options)

    return driver



def get_texts(url):

    driver = start_script()

    driver.get(url)

    if "index?continue" not in driver.current_url:

        for item in [items.text for items in driver.find_elements_by_tag_name("h3")]:

            print(item)

    else:

        get_texts(url)



if __name__ == '__main__':

    get_texts(link)

The code below works well for me, however it can't help you with bad proxies. It also loops through the list of proxies and tries one until it succeeds or the list runs out.
It prints which proxy it uses so that you can see that it tries more than one time.

However as https://www.us-proxy.org/ points out:

What is Google proxy? Proxies that support searching on Google are
called Google proxy. Some programs need them to make large number of
queries on Google. Since year 2016, all the Google proxies are dead.
Read that article for more information.

Article:

Google Blocks Proxy in 2016 Google shows a page to verify that you are
a human instead of the robot if a proxy is detected. Before the year
2016, Google allows using that proxy for some time if you can pass
this human verification.

from contextlib import contextmanager

import random



from bs4 import BeautifulSoup

import requests

from selenium import webdriver





def get_proxies():   

    response = requests.get('https://www.us-proxy.org/')

    soup = BeautifulSoup(response.text,"lxml")

    proxies = [':'.join([item.select_one("td").text,item.select_one("td:nth-of-type(2)").text]) for item in soup.select("table.table tbody tr") if "yes" in item.text]

    random.shuffle(proxies)

    return proxies





# Only need to fetch the proxies once

PROXIES = get_proxies()





@contextmanager

def proxy_driver():

    try:

        proxy = PROXIES.pop()

        print(f'Running with proxy {proxy}')

        chrome_options = webdriver.ChromeOptions()

        # chrome_options.add_argument("--headless")

        chrome_options.add_argument(f'--proxy-server={proxy}')

        driver = webdriver.Chrome(options=chrome_options)

        yield driver

    finally:

        driver.close()



def get_texts(url):

    with proxy_driver() as driver:

        driver.get(url)

        if "index?continue" not in driver.current_url:

            return [items.text for items in driver.find_elements_by_tag_name("h3")]

        print('recaptcha')



if __name__ == '__main__':

    link = 'http://www.google.com/search?q=python'

    while True:

        links = get_texts(link)

        if links:

            break

    print(links)

while True:

  driver = start_script()

  driver.get(url)

  if "index?continue" in driver.current_url:

    continue

  else:

    break

This will loop until index?continue is not in the url, and then break out of the loop.

This answer only addresses your specific question - it doesn't address the problem that you might be creating a large number of web drivers, but you never destroy the unsused / failed ones. Hint: you should.