Python code scraping data from kickstarter does not work after some iteration
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I try to scrap data from kickstarter, the code is working but it gives the following error in page 15 (you might get get error in different page since webpage is dynamic):
Traceback (most recent call last): File "C:Userslenovokick.py",
line 30, in
csvwriter.writerow(row) File "C:UserslenovoAppDataLocalProgramsPythonPython37libencodingscp1252.py",
line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character 'uff5c' in
position 27: character maps to
What might be the issue? Any suggestion?
from urllib.request import urlopen
from bs4 import BeautifulSoup
import json
import csv
KICKSTARTER_SEARCH_URL = "https://www.kickstarter.com/discover/advanced?category_id=16&sort=newest&seed=2502593&page={}"
DATA_FILE = "kickstarter.csv"
csvfile = open(DATA_FILE, 'w')
csvwriter = csv.writer(csvfile, delimiter=',')
page_start = 0
while True:
url = KICKSTARTER_SEARCH_URL.format(page_start)
print(url)
response = urlopen(url)
html = response.read()
soup = BeautifulSoup(html, 'html.parser')
project_details_divs = soup.findAll('div', {"class":"js-react-proj-card"})
if len(project_details_divs) == 0:
break;
for div in project_details_divs:
project = json.loads(div['data-project'])
row = [project["id"],project["name"],project["goal"],project["pledged"]]
csvwriter.writerow(row)
page_start +=1
csvfile.close()
python json web-scraping beautifulsoup kickstarter
edited Jan 4 at 1:00
Dale Burrell
3,46852655
asked Jan 4 at 0:34
user229519user229519
345
add a comment |
I try to scrap data from kickstarter, the code is working but it gives the following error in page 15 (you might get get error in different page since webpage is dynamic):
Traceback (most recent call last): File "C:Userslenovokick.py",
line 30, in
csvwriter.writerow(row) File "C:UserslenovoAppDataLocalProgramsPythonPython37libencodingscp1252.py",
line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character 'uff5c' in
position 27: character maps to
What might be the issue? Any suggestion?
from urllib.request import urlopen
from bs4 import BeautifulSoup
import json
import csv
KICKSTARTER_SEARCH_URL = "https://www.kickstarter.com/discover/advanced?category_id=16&sort=newest&seed=2502593&page={}"
DATA_FILE = "kickstarter.csv"
csvfile = open(DATA_FILE, 'w')
csvwriter = csv.writer(csvfile, delimiter=',')
page_start = 0
while True:
url = KICKSTARTER_SEARCH_URL.format(page_start)
print(url)
response = urlopen(url)
html = response.read()
soup = BeautifulSoup(html, 'html.parser')
project_details_divs = soup.findAll('div', {"class":"js-react-proj-card"})
if len(project_details_divs) == 0:
break;
for div in project_details_divs:
project = json.loads(div['data-project'])
row = [project["id"],project["name"],project["goal"],project["pledged"]]
csvwriter.writerow(row)
page_start +=1
csvfile.close()
python json web-scraping beautifulsoup kickstarter
edited Jan 4 at 1:00
Dale Burrell
3,46852655
asked Jan 4 at 0:34
user229519user229519
345
If any answer does what you need, do not forget to tick as correct. I remind you this because newcomers often forget to do so.
– keepAlive
Jan 4 at 0:58
add a comment |
I try to scrap data from kickstarter, the code is working but it gives the following error in page 15 (you might get get error in different page since webpage is dynamic):
Traceback (most recent call last): File "C:Userslenovokick.py",
line 30, in
csvwriter.writerow(row) File "C:UserslenovoAppDataLocalProgramsPythonPython37libencodingscp1252.py",
line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character 'uff5c' in
position 27: character maps to
What might be the issue? Any suggestion?
from urllib.request import urlopen
from bs4 import BeautifulSoup
import json
import csv
KICKSTARTER_SEARCH_URL = "https://www.kickstarter.com/discover/advanced?category_id=16&sort=newest&seed=2502593&page={}"
DATA_FILE = "kickstarter.csv"
csvfile = open(DATA_FILE, 'w')
csvwriter = csv.writer(csvfile, delimiter=',')
page_start = 0
while True:
url = KICKSTARTER_SEARCH_URL.format(page_start)
print(url)
response = urlopen(url)
html = response.read()
soup = BeautifulSoup(html, 'html.parser')
project_details_divs = soup.findAll('div', {"class":"js-react-proj-card"})
if len(project_details_divs) == 0:
break;
for div in project_details_divs:
project = json.loads(div['data-project'])
row = [project["id"],project["name"],project["goal"],project["pledged"]]
csvwriter.writerow(row)
page_start +=1
csvfile.close()
python json web-scraping beautifulsoup kickstarter
edited Jan 4 at 1:00
Dale Burrell
3,46852655
asked Jan 4 at 0:34
user229519user229519
345
I try to scrap data from kickstarter, the code is working but it gives the following error in page 15 (you might get get error in different page since webpage is dynamic):
Traceback (most recent call last): File "C:Userslenovokick.py",
line 30, in
csvwriter.writerow(row) File "C:UserslenovoAppDataLocalProgramsPythonPython37libencodingscp1252.py",
line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character 'uff5c' in
position 27: character maps to
What might be the issue? Any suggestion?
from urllib.request import urlopen
from bs4 import BeautifulSoup
import json
import csv
KICKSTARTER_SEARCH_URL = "https://www.kickstarter.com/discover/advanced?category_id=16&sort=newest&seed=2502593&page={}"
DATA_FILE = "kickstarter.csv"
csvfile = open(DATA_FILE, 'w')
csvwriter = csv.writer(csvfile, delimiter=',')
page_start = 0
while True:
url = KICKSTARTER_SEARCH_URL.format(page_start)
print(url)
response = urlopen(url)
html = response.read()
soup = BeautifulSoup(html, 'html.parser')
project_details_divs = soup.findAll('div', {"class":"js-react-proj-card"})
if len(project_details_divs) == 0:
break;
for div in project_details_divs:
project = json.loads(div['data-project'])
row = [project["id"],project["name"],project["goal"],project["pledged"]]
csvwriter.writerow(row)
page_start +=1
csvfile.close()
python json web-scraping beautifulsoup kickstarter
python json web-scraping beautifulsoup kickstarter
edited Jan 4 at 1:00
Dale Burrell
3,46852655
asked Jan 4 at 0:34
user229519user229519
345
edited Jan 4 at 1:00
Dale Burrell
3,46852655
asked Jan 4 at 0:34
user229519user229519
345
edited Jan 4 at 1:00
Dale Burrell
3,46852655
edited Jan 4 at 1:00
Dale Burrell
3,46852655
edited Jan 4 at 1:00
Dale Burrell
3,46852655
3,46852655
asked Jan 4 at 0:34
user229519user229519
345
asked Jan 4 at 0:34
user229519user229519
345
asked Jan 4 at 0:34
user229519user229519
345
345
If any answer does what you need, do not forget to tick as correct. I remind you this because newcomers often forget to do so.
– keepAlive
Jan 4 at 0:58
add a comment |
If any answer does what you need, do not forget to tick as correct. I remind you this because newcomers often forget to do so.
– keepAlive
Jan 4 at 0:58
If any answer does what you need, do not forget to tick as correct. I remind you this because newcomers often forget to do so.
– keepAlive
Jan 4 at 0:58
If any answer does what you need, do not forget to tick as correct. I remind you this because newcomers often forget to do so.
– keepAlive
Jan 4 at 0:58
add a comment |
1 Answer
1
active
oldest
votes
Add the argument encoding to your file-opener. I mean, change
csvfile = open(DATA_FILE, 'w')
into
csvfile = open(DATA_FILE, 'w', encoding='utf-8')
answered Jan 4 at 0:50
keepAlivekeepAlive
3,22041224
Any question @user229 ?
– keepAlive
Jan 4 at 0:56
It gives the error: TypeError: 'encoding' is an invalid keyword argument for this function
– user229519
Jan 4 at 0:58
@user229, see my update. Does it work?
– keepAlive
Jan 4 at 1:03
Should I also change file writing? because it again gives error inside for loop, but in the page 1.
– user229519
Jan 4 at 1:07
1
my bad, it works, thanks a lot.
– user229519
Jan 4 at 1:13
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54031714%2fpython-code-scraping-data-from-kickstarter-does-not-work-after-some-iteration%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Add the argument encoding to your file-opener. I mean, change
csvfile = open(DATA_FILE, 'w')
into
csvfile = open(DATA_FILE, 'w', encoding='utf-8')
answered Jan 4 at 0:50
keepAlivekeepAlive
3,22041224
Any question @user229 ?
– keepAlive
Jan 4 at 0:56
It gives the error: TypeError: 'encoding' is an invalid keyword argument for this function
– user229519
Jan 4 at 0:58
@user229, see my update. Does it work?
– keepAlive
Jan 4 at 1:03
Should I also change file writing? because it again gives error inside for loop, but in the page 1.
– user229519
Jan 4 at 1:07
1
my bad, it works, thanks a lot.
– user229519
Jan 4 at 1:13
add a comment |
Add the argument encoding to your file-opener. I mean, change
csvfile = open(DATA_FILE, 'w')
into
csvfile = open(DATA_FILE, 'w', encoding='utf-8')
answered Jan 4 at 0:50
keepAlivekeepAlive
3,22041224
Any question @user229 ?
– keepAlive
Jan 4 at 0:56
It gives the error: TypeError: 'encoding' is an invalid keyword argument for this function
– user229519
Jan 4 at 0:58
@user229, see my update. Does it work?
– keepAlive
Jan 4 at 1:03
Should I also change file writing? because it again gives error inside for loop, but in the page 1.
– user229519
Jan 4 at 1:07
1
my bad, it works, thanks a lot.
– user229519
Jan 4 at 1:13
add a comment |
Add the argument encoding to your file-opener. I mean, change
csvfile = open(DATA_FILE, 'w')
into
csvfile = open(DATA_FILE, 'w', encoding='utf-8')
answered Jan 4 at 0:50
keepAlivekeepAlive
3,22041224
Add the argument encoding to your file-opener. I mean, change
csvfile = open(DATA_FILE, 'w')
into
csvfile = open(DATA_FILE, 'w', encoding='utf-8')
answered Jan 4 at 0:50
keepAlivekeepAlive
3,22041224
edited Jan 4 at 1:02
answered Jan 4 at 0:50
keepAlivekeepAlive
3,22041224
answered Jan 4 at 0:50
keepAlivekeepAlive
3,22041224
answered Jan 4 at 0:50
keepAlivekeepAlive
3,22041224
3,22041224
Any question @user229 ?
– keepAlive
Jan 4 at 0:56
It gives the error: TypeError: 'encoding' is an invalid keyword argument for this function
– user229519
Jan 4 at 0:58
@user229, see my update. Does it work?
– keepAlive
Jan 4 at 1:03
Should I also change file writing? because it again gives error inside for loop, but in the page 1.
– user229519
Jan 4 at 1:07
1
my bad, it works, thanks a lot.
– user229519
Jan 4 at 1:13
add a comment |
Any question @user229 ?
– keepAlive
Jan 4 at 0:56
It gives the error: TypeError: 'encoding' is an invalid keyword argument for this function
– user229519
Jan 4 at 0:58
@user229, see my update. Does it work?
– keepAlive
Jan 4 at 1:03
Should I also change file writing? because it again gives error inside for loop, but in the page 1.
– user229519
Jan 4 at 1:07
1
my bad, it works, thanks a lot.
– user229519
Jan 4 at 1:13
Any question @user229 ?
– keepAlive
Jan 4 at 0:56
Any question @user229 ?
– keepAlive
Jan 4 at 0:56
It gives the error: TypeError: 'encoding' is an invalid keyword argument for this function
– user229519
Jan 4 at 0:58
It gives the error: TypeError: 'encoding' is an invalid keyword argument for this function
– user229519
Jan 4 at 0:58
@user229, see my update. Does it work?
– keepAlive
Jan 4 at 1:03
@user229, see my update. Does it work?
– keepAlive
Jan 4 at 1:03
Should I also change file writing? because it again gives error inside for loop, but in the page 1.
– user229519
Jan 4 at 1:07
Should I also change file writing? because it again gives error inside for loop, but in the page 1.
– user229519
Jan 4 at 1:07
1
1
my bad, it works, thanks a lot.
– user229519
Jan 4 at 1:13
my bad, it works, thanks a lot.
– user229519
Jan 4 at 1:13
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54031714%2fpython-code-scraping-data-from-kickstarter-does-not-work-after-some-iteration%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
If any answer does what you need, do not forget to tick as correct. I remind you this because newcomers often forget to do so.
– keepAlive
Jan 4 at 0:58