Parsing tables from the site
There is a site https://ru.myip.ms/browse/market_bitcoin/%D0%91%D0%B8%D1%82%D0%BA%D0%BE%D0%B8%D0%BD_%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D1%8F_%D1%86%D0%B5%D0%BD.html#a, below is a table with BTC prices, I need to like then parse this table. I was trying to do, but for some reason, instead of the price in the table is displayed dots
from time import sleep
import pandas as pd
import requests
host = 'ru.myip.ms'
index_url = 'https://ru.myip.ms'
home_url = "https://ru.myip.ms/browse/market_bitcoin/%D0%91%D0%B8%D1%82%D0%BA%D0%BE%D0%B8%D0%BD_%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D1%8F_%D1%86%D0%B5%D0%BD.html#a"
base_ajax_url = "https://ru.myip.ms/ajax_table/market_bitcoin/{page}"
with requests.Session() as session:
session.headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
'Host': host
}
# visit home page and parse the initial dataframe
response = session.get(home_url)
df = pd.read_html(response.text, attrs={"id": "market_bitcoin_tbl"})[0]
df = df.rename(columns=lambda x: x.strip()) # remove extra newlines from the column names
sleep(2)
# start paginating with page=2
page = 1
while True:
url = base_ajax_url.format(page=page)
print("Processing {url}...".format(url=url))
response = session.post(url,
data={'getpage': 'yes', 'lang': 'ru'},
headers={
'X-Requested-With': 'XMLHttpRequest',
'Origin': index_url,
'Referer': home_url
})
# add data to the existing dataframe
try:
new_df = pd.read_html("<table>{0}</table>".format(response.text))[0]
except ValueError: # could not extract data from HTML - last page?
break
new_df.columns = df.columns
df = pd.concat([df, new_df])
page += 1
sleep(1)
print(df)
python parsing
asked Jan 1 at 21:33
AyurpwnzAyurpwnz
61
add a comment |
There is a site https://ru.myip.ms/browse/market_bitcoin/%D0%91%D0%B8%D1%82%D0%BA%D0%BE%D0%B8%D0%BD_%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D1%8F_%D1%86%D0%B5%D0%BD.html#a, below is a table with BTC prices, I need to like then parse this table. I was trying to do, but for some reason, instead of the price in the table is displayed dots
from time import sleep
import pandas as pd
import requests
host = 'ru.myip.ms'
index_url = 'https://ru.myip.ms'
home_url = "https://ru.myip.ms/browse/market_bitcoin/%D0%91%D0%B8%D1%82%D0%BA%D0%BE%D0%B8%D0%BD_%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D1%8F_%D1%86%D0%B5%D0%BD.html#a"
base_ajax_url = "https://ru.myip.ms/ajax_table/market_bitcoin/{page}"
with requests.Session() as session:
session.headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
'Host': host
}
# visit home page and parse the initial dataframe
response = session.get(home_url)
df = pd.read_html(response.text, attrs={"id": "market_bitcoin_tbl"})[0]
df = df.rename(columns=lambda x: x.strip()) # remove extra newlines from the column names
sleep(2)
# start paginating with page=2
page = 1
while True:
url = base_ajax_url.format(page=page)
print("Processing {url}...".format(url=url))
response = session.post(url,
data={'getpage': 'yes', 'lang': 'ru'},
headers={
'X-Requested-With': 'XMLHttpRequest',
'Origin': index_url,
'Referer': home_url
})
# add data to the existing dataframe
try:
new_df = pd.read_html("<table>{0}</table>".format(response.text))[0]
except ValueError: # could not extract data from HTML - last page?
break
new_df.columns = df.columns
df = pd.concat([df, new_df])
page += 1
sleep(1)
print(df)
python parsing
asked Jan 1 at 21:33
AyurpwnzAyurpwnz
61
Who knows how to fix this?
– Ayurpwnz
Jan 1 at 21:45
add a comment |
There is a site https://ru.myip.ms/browse/market_bitcoin/%D0%91%D0%B8%D1%82%D0%BA%D0%BE%D0%B8%D0%BD_%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D1%8F_%D1%86%D0%B5%D0%BD.html#a, below is a table with BTC prices, I need to like then parse this table. I was trying to do, but for some reason, instead of the price in the table is displayed dots
from time import sleep
import pandas as pd
import requests
host = 'ru.myip.ms'
index_url = 'https://ru.myip.ms'
home_url = "https://ru.myip.ms/browse/market_bitcoin/%D0%91%D0%B8%D1%82%D0%BA%D0%BE%D0%B8%D0%BD_%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D1%8F_%D1%86%D0%B5%D0%BD.html#a"
base_ajax_url = "https://ru.myip.ms/ajax_table/market_bitcoin/{page}"
with requests.Session() as session:
session.headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
'Host': host
}
# visit home page and parse the initial dataframe
response = session.get(home_url)
df = pd.read_html(response.text, attrs={"id": "market_bitcoin_tbl"})[0]
df = df.rename(columns=lambda x: x.strip()) # remove extra newlines from the column names
sleep(2)
# start paginating with page=2
page = 1
while True:
url = base_ajax_url.format(page=page)
print("Processing {url}...".format(url=url))
response = session.post(url,
data={'getpage': 'yes', 'lang': 'ru'},
headers={
'X-Requested-With': 'XMLHttpRequest',
'Origin': index_url,
'Referer': home_url
})
# add data to the existing dataframe
try:
new_df = pd.read_html("<table>{0}</table>".format(response.text))[0]
except ValueError: # could not extract data from HTML - last page?
break
new_df.columns = df.columns
df = pd.concat([df, new_df])
page += 1
sleep(1)
print(df)
python parsing
asked Jan 1 at 21:33
AyurpwnzAyurpwnz
61
There is a site https://ru.myip.ms/browse/market_bitcoin/%D0%91%D0%B8%D1%82%D0%BA%D0%BE%D0%B8%D0%BD_%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D1%8F_%D1%86%D0%B5%D0%BD.html#a, below is a table with BTC prices, I need to like then parse this table. I was trying to do, but for some reason, instead of the price in the table is displayed dots
from time import sleep
import pandas as pd
import requests
host = 'ru.myip.ms'
index_url = 'https://ru.myip.ms'
home_url = "https://ru.myip.ms/browse/market_bitcoin/%D0%91%D0%B8%D1%82%D0%BA%D0%BE%D0%B8%D0%BD_%D0%B8%D1%81%D1%82%D0%BE%D1%80%D0%B8%D1%8F_%D1%86%D0%B5%D0%BD.html#a"
base_ajax_url = "https://ru.myip.ms/ajax_table/market_bitcoin/{page}"
with requests.Session() as session:
session.headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36',
'Host': host
}
# visit home page and parse the initial dataframe
response = session.get(home_url)
df = pd.read_html(response.text, attrs={"id": "market_bitcoin_tbl"})[0]
df = df.rename(columns=lambda x: x.strip()) # remove extra newlines from the column names
sleep(2)
# start paginating with page=2
page = 1
while True:
url = base_ajax_url.format(page=page)
print("Processing {url}...".format(url=url))
response = session.post(url,
data={'getpage': 'yes', 'lang': 'ru'},
headers={
'X-Requested-With': 'XMLHttpRequest',
'Origin': index_url,
'Referer': home_url
})
# add data to the existing dataframe
try:
new_df = pd.read_html("<table>{0}</table>".format(response.text))[0]
except ValueError: # could not extract data from HTML - last page?
break
new_df.columns = df.columns
df = pd.concat([df, new_df])
page += 1
sleep(1)
print(df)
python parsing
python parsing
asked Jan 1 at 21:33
AyurpwnzAyurpwnz
61
asked Jan 1 at 21:33
AyurpwnzAyurpwnz
61
asked Jan 1 at 21:33
AyurpwnzAyurpwnz
61
asked Jan 1 at 21:33
AyurpwnzAyurpwnz
61
asked Jan 1 at 21:33
AyurpwnzAyurpwnz
61
61
Who knows how to fix this?
– Ayurpwnz
Jan 1 at 21:45
add a comment |
Who knows how to fix this?
– Ayurpwnz
Jan 1 at 21:45
Who knows how to fix this?
– Ayurpwnz
Jan 1 at 21:45
Who knows how to fix this?
– Ayurpwnz
Jan 1 at 21:45
add a comment |
1 Answer
1
active
oldest
votes
you are doing it correctly. and you already have your results.
try just to do this to see the result.
print(df['Bitcoin Price'])
you see the dots, just because the df is big to show it all when you run it, but it exists.
answered Jan 1 at 22:06
Yasi EsYasi Es
1
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53999112%2fparsing-tables-from-the-site%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
you are doing it correctly. and you already have your results.
try just to do this to see the result.
print(df['Bitcoin Price'])
you see the dots, just because the df is big to show it all when you run it, but it exists.
answered Jan 1 at 22:06
Yasi EsYasi Es
1
add a comment |
you are doing it correctly. and you already have your results.
try just to do this to see the result.
print(df['Bitcoin Price'])
you see the dots, just because the df is big to show it all when you run it, but it exists.
answered Jan 1 at 22:06
Yasi EsYasi Es
1
add a comment |
you are doing it correctly. and you already have your results.
try just to do this to see the result.
print(df['Bitcoin Price'])
you see the dots, just because the df is big to show it all when you run it, but it exists.
answered Jan 1 at 22:06
Yasi EsYasi Es
1
you are doing it correctly. and you already have your results.
try just to do this to see the result.
print(df['Bitcoin Price'])
you see the dots, just because the df is big to show it all when you run it, but it exists.
answered Jan 1 at 22:06
Yasi EsYasi Es
1
answered Jan 1 at 22:06
Yasi EsYasi Es
1
answered Jan 1 at 22:06
Yasi EsYasi Es
1
answered Jan 1 at 22:06
Yasi EsYasi Es
1
1
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53999112%2fparsing-tables-from-the-site%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Who knows how to fix this?
– Ayurpwnz
Jan 1 at 21:45