Extract data using Beautiful Soup and Requests












1















I am trying to scrape data from Stackoverflow using Beautiful soup and requests package in Python. I have been able to extract most of the details, however when I try to extract the reputation scores of an user I am only able to pull data for reputation score and Gold, but not able to extract data for Silver and Bronze counts.



Given below is the code I am using to extract:



from bs4 import BeautifulSoup

import requests



source = requests.get('https://stackoverflow.com/questions/53968946/how-can-i-limit-function-slot-play-just-for-5-turn-with-do-while-loop').text

soup = BeautifulSoup(source,'lxml')

article = soup.find('div', class_='inner-content clearfix')

user_reputation_score = article.find('span', class_='reputation-score').text

print(user_reputation_score)


Code for Gold badge:



gold_badge = article.find('div', class_='-flair').find('span', class_='badgecount').text

print(gold_badge)


Wondering how to extend the above to extract data for Silver and Bronze batches.



Use the below link to test:



https://stackoverflow.com/questions/53968946/how-can-i-limit-function-slot-play-just-for-5-turn-with-do-while-loop


Please note I am doing this for pure educational purpose. Thanks.






python beautifulsoup python-requests







share|improve this question





























    1















    I am trying to scrape data from Stackoverflow using Beautiful soup and requests package in Python. I have been able to extract most of the details, however when I try to extract the reputation scores of an user I am only able to pull data for reputation score and Gold, but not able to extract data for Silver and Bronze counts.



    Given below is the code I am using to extract:



    from bs4 import BeautifulSoup
    
import requests
    

    
source = requests.get('https://stackoverflow.com/questions/53968946/how-can-i-limit-function-slot-play-just-for-5-turn-with-do-while-loop').text
    
soup = BeautifulSoup(source,'lxml')
    
article = soup.find('div', class_='inner-content clearfix')
    
user_reputation_score = article.find('span', class_='reputation-score').text
    
print(user_reputation_score)


    Code for Gold badge:



    gold_badge = article.find('div', class_='-flair').find('span', class_='badgecount').text
    
print(gold_badge)


    Wondering how to extend the above to extract data for Silver and Bronze batches.



    Use the below link to test:



    https://stackoverflow.com/questions/53968946/how-can-i-limit-function-slot-play-just-for-5-turn-with-do-while-loop


    Please note I am doing this for pure educational purpose. Thanks.






    python beautifulsoup python-requests







    share|improve this question



























      1












      1








      1








      I am trying to scrape data from Stackoverflow using Beautiful soup and requests package in Python. I have been able to extract most of the details, however when I try to extract the reputation scores of an user I am only able to pull data for reputation score and Gold, but not able to extract data for Silver and Bronze counts.



      Given below is the code I am using to extract:



      from bs4 import BeautifulSoup
      
import requests
      

      
source = requests.get('https://stackoverflow.com/questions/53968946/how-can-i-limit-function-slot-play-just-for-5-turn-with-do-while-loop').text
      
soup = BeautifulSoup(source,'lxml')
      
article = soup.find('div', class_='inner-content clearfix')
      
user_reputation_score = article.find('span', class_='reputation-score').text
      
print(user_reputation_score)


      Code for Gold badge:



      gold_badge = article.find('div', class_='-flair').find('span', class_='badgecount').text
      
print(gold_badge)


      Wondering how to extend the above to extract data for Silver and Bronze batches.



      Use the below link to test:



      https://stackoverflow.com/questions/53968946/how-can-i-limit-function-slot-play-just-for-5-turn-with-do-while-loop


      Please note I am doing this for pure educational purpose. Thanks.






      python beautifulsoup python-requests







      share|improve this question
















      I am trying to scrape data from Stackoverflow using Beautiful soup and requests package in Python. I have been able to extract most of the details, however when I try to extract the reputation scores of an user I am only able to pull data for reputation score and Gold, but not able to extract data for Silver and Bronze counts.



      Given below is the code I am using to extract:



      from bs4 import BeautifulSoup
      
import requests
      

      
source = requests.get('https://stackoverflow.com/questions/53968946/how-can-i-limit-function-slot-play-just-for-5-turn-with-do-while-loop').text
      
soup = BeautifulSoup(source,'lxml')
      
article = soup.find('div', class_='inner-content clearfix')
      
user_reputation_score = article.find('span', class_='reputation-score').text
      
print(user_reputation_score)


      Code for Gold badge:



      gold_badge = article.find('div', class_='-flair').find('span', class_='badgecount').text
      
print(gold_badge)


      Wondering how to extend the above to extract data for Silver and Bronze batches.



      Use the below link to test:



      https://stackoverflow.com/questions/53968946/how-can-i-limit-function-slot-play-just-for-5-turn-with-do-while-loop


      Please note I am doing this for pure educational purpose. Thanks.






      python beautifulsoup python-requests




      python beautifulsoup python-requests






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Dec 29 '18 at 23:20









      ewwink

      11.6k22238




      11.6k22238










      asked Dec 29 '18 at 16:15









      dark horsedark horse

      14810




      14810
























          1 Answer
          1






          active

          oldest

          votes


















          1














          find() return first element, to get multiple element use find_all()



          badge = article.find('div', class_='-flair').find_all('span', class_='badgecount')
          
gold_badge = badge[0].text
          
silver_badge = badge[1].text
          
bronze_badge = badge[2].text
          
print(gold_badge, silver_badge, bronze_badge) # 2 7 26





          share|improve this answer
























          • thanks that helped..

            – dark horse
            Jan 2 at 11:10











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53971195%2fextract-data-using-beautiful-soup-and-requests%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          1














          find() return first element, to get multiple element use find_all()



          badge = article.find('div', class_='-flair').find_all('span', class_='badgecount')
          
gold_badge = badge[0].text
          
silver_badge = badge[1].text
          
bronze_badge = badge[2].text
          
print(gold_badge, silver_badge, bronze_badge) # 2 7 26





          share|improve this answer
























          • thanks that helped..

            – dark horse
            Jan 2 at 11:10
















          1














          find() return first element, to get multiple element use find_all()



          badge = article.find('div', class_='-flair').find_all('span', class_='badgecount')
          
gold_badge = badge[0].text
          
silver_badge = badge[1].text
          
bronze_badge = badge[2].text
          
print(gold_badge, silver_badge, bronze_badge) # 2 7 26





          share|improve this answer
























          • thanks that helped..

            – dark horse
            Jan 2 at 11:10














          1












          1








          1







          find() return first element, to get multiple element use find_all()



          badge = article.find('div', class_='-flair').find_all('span', class_='badgecount')
          
gold_badge = badge[0].text
          
silver_badge = badge[1].text
          
bronze_badge = badge[2].text
          
print(gold_badge, silver_badge, bronze_badge) # 2 7 26





          share|improve this answer













          find() return first element, to get multiple element use find_all()



          badge = article.find('div', class_='-flair').find_all('span', class_='badgecount')
          
gold_badge = badge[0].text
          
silver_badge = badge[1].text
          
bronze_badge = badge[2].text
          
print(gold_badge, silver_badge, bronze_badge) # 2 7 26






          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Dec 29 '18 at 17:31









          ewwinkewwink

          11.6k22238




          11.6k22238













          • thanks that helped..

            – dark horse
            Jan 2 at 11:10



















          • thanks that helped..

            – dark horse
            Jan 2 at 11:10

















          thanks that helped..

          – dark horse
          Jan 2 at 11:10





          thanks that helped..

          – dark horse
          Jan 2 at 11:10


















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53971195%2fextract-data-using-beautiful-soup-and-requests%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          Monofisismo

          Image
          Monofisismo (do grego: monos - "único, singular" e physis - "natureza") é o ponto de vista cristológico que defende que, depois da união do divino e do humano na encarnação histórica, Jesus Cristo, como encarnação do Filho ou Verbo ( Logos ) de Deus, teria apenas uma única "natureza", a divina, e não uma síntese de ambas. O monofisismo é contraposto pelo diofisismo (ou "diafisismo"), que defende que Jesus preservou em si as duas naturezas. Historicamente, o monofisismo se refere primordialmente à posição dos que (especialmente no Egito e, em menor grau, na Síria) rejeitaram as decisões do Concílio de Calcedônia em 451 (o quarto concílio ecumênico). Os membros mais moderados entre eles, porém, defendem a teologia "miafisita", que se tornou a oficial para as Igrejas Ortodoxas Orientais. Muitos ortodoxos orientais, porém, rejeitam essa classificação, mesmo como um termo genérico, mas ele é amplamente utilizado na literatura históri
          Read more

          Angular Downloading a file using contenturl with Basic Authentication

          Image
          0 I have a contentUrl when user clicks on that,file should be downloaded. If I try to browse the contentUrl in the browser it asks for Userid and password.I believe this is Basic Authentication.I dont want user to enter UserId and Password so I am passing it Authorization(Please see below code). Url looks some thing like this: http://XXX.XXX.XX.XXX:8080/flowable-rest/service/runtime/tasks/90875/attachments/90555/content let username : string = 'admin'; let pwd : string = 'test'; let headers = new Headers(); headers.set('Accept', 'text/json'); headers.append('Content-Type', 'application/json'); headers.set('Access-Control-Allow-Origin', '*'); headers.set('Access-Control-Allow-Methods','POST, GET, OPTIONS, PUT, DELETE'); head
          Read more

          Olmecas

          Image
          Monumento 1, uma das quatro cabeças colossais encontradas em La Venta, com altura aproximada de 3 metros Olmecas é a designação do povo e da civilização que estiveram na origem da antiga cultura pré-colombiana da Mesoamérica e que se desenvolveram nas regiões tropicais do centro-sul do atual México durante o pré-clássico, próximo de onde hoje estão localizados os estados mexicanos de Veracruz e Tabasco, no Istmo de Tehuantepec, numa zona designada área nuclear olmeca. A cultura olmeca floresceu nesta região aproximadamente entre 1500 e 400 a.C., [ 1 ] e crê-se que tenha sido a civilização-mãe de todas as civilizações mesoamericanas que se desenvolveram posteriormente. [ 2 ] No entanto, desconhece-se a sua exacta filiação étnica, ainda que existam numerosas hipóteses colocadas para tentar resolver esta questão. O etnónimo olmeca foi cunhado pelos arqueólogos do século XX, e não devem confundir-se com os muito posteriores olmecas-xicalancas que ocuparam vários locais do México
          Read more