Bdtjtk

Found the solution. please see revised code that works. I have reviewed other similar issues. I think mine is different in that I first go to the known website and then grab the links to the individual files and save to a list, then retrieve the files one after another. I am building code that will take the URL of a website then scan that webenter code heresite for needed links for work to mass download the files instead of clicking on each link and download one by one for a recurring end of the month task. Those topic links are saved into a list, univ_list. I am using a while loop.

There are 164 links that need to be downloaded for this month. I use a join with the base URL, http://www.linksneeded.com/ + the file piece (since each file has its own piece of the puzzle for example, This is the first file.xls) so it would look like this: http://www.linksneeded.com/This is the first file.xls. The beginning of each file full location remains the same while the individual file names change that I have saved in univ_list. I need the first pass to use univ_list[1], the second to use univ_list[2] ..... all the way to the 164 (and that number changes per site and each month) univ_list[164].

I have tried setting another variable and performing a += 1 with each loop but get invalid syntax for this attempt: print(page_retrieve + univ_list += 1). I have it set to print to validate the output.

import urllib

from urllib.request import urlopen

from bs4 import BeautifulSoup

import requests



page_retrieve = input('Enter page URL to retrieve: ')



page = requests.get(page_retrieve)

type(page)



page.status_code == requests.codes.ok

print()



print('Successful page access:', page.status_code == requests.codes.ok)

links = 1

print()



if page.status_code != 200:

    print('404 Client Error: URL Not Found for: ', (page_retrieve))

    print('Data file non existent. Please review file URL for accuracy.')



if page.status_code == 200:

    univ_list = 

    html = urlopen(page_retrieve)

    bsObj = BeautifulSoup(html, "html.parser")

    for link in bsObj.findAll("a"):

        if 'href' in link.attrs:

            print(link.attrs['href'], 'n')

            univ_list.append(link.attrs['href'])



while links < len(univ_list):

    #print(page_retrieve + univ_list[-1])

    links = links + 1

    #print(links)



#Actual download of files

while links > 0:

    print('Number of Links left to process:', links)

    #print('Full file URL:', page_retrieve + univ_list[-1])

    #print()

    fileURL = ('page_retrieve + univ_list[-1]')

    #print(univ_list[-1])

    print()

    urllib.request.urlretrieve(page_retrieve + univ_list[-1], univ_list[-1])

    print('Link to download:', univ_list[-1])

    print()

    univ_list.pop()

    print('Link Removed:', univ_list[-1])

    print()

    links = links - 1

    #print('Number of Links left to process:', links)

    #print()



print()

print('Code complete')

I get the full path to the file for 2nd file just to show my code works to thatenter code here point, but cannot figure how to increment the univ_list list value to go through each link pulled. Once I can get the univ_list values increment I am going to add the retrieve piece using a modified open, write and close method I developed that I know works in another piece of code I have developed. All in an effort to automate downloading of files on several websites for end of the month data pulls that I then use for reporting compliance progress for my company policies. I have tried saving the links to a text file but get only one line with all 164 links, so looked at lists and now having this problem.