Bdtjtk

It seems impossible or very complicated to keep the original elements in webdriver of selenium after moving another page via a link generated by javascript. How can I do this?

I'm trying to do web scraping for a particular web page using the following components:

• Ubuntu 18.04.1 LTS


• Python 3.6.1


• Selenium (Python package) 3.141.0


• Google Chrome 71.0.3578.98


• ChromeDriver 2.45.615279

The web page includes links which "href" is javascript function like the following:

<a href="javascript:funcName(10, 24, 100)"></a>

The definition of the function is something like this.

var funcName = function(arg1, arg2, arg3) {

    var url = 'XXXXXXXX' // dynamically generated using arguments

    var form = $('<form>', {

        name: 'formName',

        action: url,

        method: 'post'

    });

    // Some procedure to enhance the form element with input arguments.

    form.submit()

}

The above post request redirects me to another page which I'd like to scrape.

The thing is the original web page includes many links and I'd like to scrape redirected pages one by one. However, it seems impossible to get the redirected page's url without actually clicking the link (<a>) as it's redirected by dynamically generated post request. On the other hand, if I click it and move to the redirected page, the elements I used for the original web page cannot be used anymore, so, after coming back to the original page, I need to get the next link from the beginning. This feels very redundant.

Python code example

for a in driver.find_elements_by_css_selector(.some-class-name):

    a.click()  # this redirects me to another page

    print(driver.current_url)  # this shows the redirected page

    driver.back()

    print(driver.current_url). # this shows the original page

    # After coming back to the original page and when doing looping process, Python returns StaleElementReferenceException

    # because a is attached to the original page before redirected.

What I did to keep the original page's elements but did not work:

1.Copy a element (or driver) object

from copy import deepcopy

for a in driver.find_elements_by_css_selector(.some-class-name):

    a2 = deepcopy(a)

    a2.click()  # this redirects me to another page

    print(driver.current_url)  # Expected result is that this remains the original web page, but didn't

I tried deepcopy for driver itself, but didn't work either.
Returned error is

TypeError: can't pickle _thread.lock objects

2.Open a redirected page in a new tab

from selenium.webdriver import ActionChains

from selenium.webdriver.common.keys import Keys



for a in driver.find_elements_by_css_selector(.some-class-name):



    action = ActionChains(driver)



    # Expected result is the following open the redirected page in a new tab, and CONTROL + TAB changes between tabs

    action.key_down(Keys.CONTROL).click(a).key_down(Keys.CONTROL).perform()  

    driver.send_keys(Keys.CONTROL + Keys.TAB)

However, this didn't open a new tab, just move to the redirected page in the same tab.

If there is no simple way, I will do this by creating a list or dictionary object to store which links I've already scraped, and every time after scraping redirected page, I'll parse the original page over again and skip the link that has already been checked. But I don't want to do because it's very redundant.

Even you return the same page, but selenium don't know it's the same page, selenium will treat it as an new page. The links found before the for loop is not belong to the new page. You need to find the links again on the new page and assign them to the same variable links inside for loop. Using index to iterate to next link.

links = driver.find_elements_by_css_selector(.some-class-name)



for i in range(0, len(links)):

    links[i].click()  # this redirects me to another page

    print(driver.current_url)  # this shows the redirected page

    driver.back()

    print(driver.current_url). 



    # Important: find the links again on the page back from redirected page

    # to resolve the StaleElementReferenceException.

    links = driver.find_elements_by_css_selector(.some-class-name)

I chose a way to create another webdriver instance.

driver = webdriver.Chrome()

driver_sub = webdriver.Chrome()



driver.get(url)

driver_sub.get(url)  # access the same page with different instance



for a in driver.find_elements_by_css_selector('.some-class-name'):

    script = a.get_attribute('href')

    driver_sub.execute_script(script)

    # do some work on the redirected page with driver_sub

    driver_sub.execute_script('window.history.go(-1)')  # this is almost same as driver_sub.back()