How to handle error raise - Asyncio in Python3












1















I've got a function f() that makes an API call and I want to call it multiple times asynchronously. I use the asyncio lib like that:



async def main():

    loop = asyncio.get_event_loop()

    futures = [loop.run_in_executor(None, f) for i in range(10)]

    await asyncio.gather(*futures)

    return futures



result = asyncio.get_event_loop().run_until_complete(main())


The problem is that sometimes f() raises an Exception and I'm not sure how to handle it. The doc says that Futures can contain an Exception, but that's not the case here, error is raised and program crashes.



How do I achieve that ? I think I could write a wrapper for f() and try: catch: the Exception, but that seems ugly if the feature is provided by the lib.



Thanks in advance for help,






python python-asyncio







share|improve this question



























    1















    I've got a function f() that makes an API call and I want to call it multiple times asynchronously. I use the asyncio lib like that:



    async def main():
    
    loop = asyncio.get_event_loop()
    
    futures = [loop.run_in_executor(None, f) for i in range(10)]
    
    await asyncio.gather(*futures)
    
    return futures
    

    
result = asyncio.get_event_loop().run_until_complete(main())


    The problem is that sometimes f() raises an Exception and I'm not sure how to handle it. The doc says that Futures can contain an Exception, but that's not the case here, error is raised and program crashes.



    How do I achieve that ? I think I could write a wrapper for f() and try: catch: the Exception, but that seems ugly if the feature is provided by the lib.



    Thanks in advance for help,






    python python-asyncio







    share|improve this question

























      1












      1








      1








      I've got a function f() that makes an API call and I want to call it multiple times asynchronously. I use the asyncio lib like that:



      async def main():
      
    loop = asyncio.get_event_loop()
      
    futures = [loop.run_in_executor(None, f) for i in range(10)]
      
    await asyncio.gather(*futures)
      
    return futures
      

      
result = asyncio.get_event_loop().run_until_complete(main())


      The problem is that sometimes f() raises an Exception and I'm not sure how to handle it. The doc says that Futures can contain an Exception, but that's not the case here, error is raised and program crashes.



      How do I achieve that ? I think I could write a wrapper for f() and try: catch: the Exception, but that seems ugly if the feature is provided by the lib.



      Thanks in advance for help,






      python python-asyncio







      share|improve this question














      I've got a function f() that makes an API call and I want to call it multiple times asynchronously. I use the asyncio lib like that:



      async def main():
      
    loop = asyncio.get_event_loop()
      
    futures = [loop.run_in_executor(None, f) for i in range(10)]
      
    await asyncio.gather(*futures)
      
    return futures
      

      
result = asyncio.get_event_loop().run_until_complete(main())


      The problem is that sometimes f() raises an Exception and I'm not sure how to handle it. The doc says that Futures can contain an Exception, but that's not the case here, error is raised and program crashes.



      How do I achieve that ? I think I could write a wrapper for f() and try: catch: the Exception, but that seems ugly if the feature is provided by the lib.



      Thanks in advance for help,






      python python-asyncio




      python python-asyncio






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Jan 3 at 12:41









      MeanStreetMeanStreet

      363418




      363418
























          1 Answer
          1






          active

          oldest

          votes


















          1















          The problem is that sometimes f() raises an Exception and I'm not sure how to handle it.




          That will depend on what you want to do when an exception occurs. Remember that asyncio.gather() is convenience API that propagates exceptions by default to avoid blindly continuing in case of error. If you want to proceed in case of exception, you have other options:




          • Pass return_exceptions=True to gather - this will cause gather to return the exception objects along with other results. Convenient and easy to use, but mixes exceptions with regular results, which is a bit messy.


          • Use asyncio.wait() instead of asyncio.gather(). It returns a set of futures, for which you can test whether they completed by raising or by producing a result.


          • Wrap f() in your own function that catches the exception as you see fit. You considered and rejected this, but in some cases it's exactly the right approach.







          share|improve this answer
























          • Thanks a lot for your answer, that's what I was looking for. I ran a few tests don't see clearly the difference between the first 2 options, it seems to me that gather(*futures, return_exceptions=True) does the same thing than wait(futures) (except that wait does not respect order), in both cases exceptions are mixed with results right ? (It's not a problem for my usecase, just to understand. Marked as solved)

            – MeanStreet
            Jan 3 at 13:27






          • 1





            @MeanStreet The difference is that wait() returns sets of futures. Given a future, you can extract its result with result(), which will either return the result or re-raise the exception if there was one. But you can also call exception() to test whether an exception was raised. This allows you to distinguish between a function that raised an exception and one that (for whatever reason) happened to return an Exception instance.

            – user4815162342
            Jan 3 at 14:21













          • Oh I got it, that's what you meant by "mixes exceptions with regular results". Thanks

            – MeanStreet
            Jan 3 at 14:33












          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54022514%2fhow-to-handle-error-raise-asyncio-in-python3%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          1















          The problem is that sometimes f() raises an Exception and I'm not sure how to handle it.




          That will depend on what you want to do when an exception occurs. Remember that asyncio.gather() is convenience API that propagates exceptions by default to avoid blindly continuing in case of error. If you want to proceed in case of exception, you have other options:




          • Pass return_exceptions=True to gather - this will cause gather to return the exception objects along with other results. Convenient and easy to use, but mixes exceptions with regular results, which is a bit messy.


          • Use asyncio.wait() instead of asyncio.gather(). It returns a set of futures, for which you can test whether they completed by raising or by producing a result.


          • Wrap f() in your own function that catches the exception as you see fit. You considered and rejected this, but in some cases it's exactly the right approach.







          share|improve this answer
























          • Thanks a lot for your answer, that's what I was looking for. I ran a few tests don't see clearly the difference between the first 2 options, it seems to me that gather(*futures, return_exceptions=True) does the same thing than wait(futures) (except that wait does not respect order), in both cases exceptions are mixed with results right ? (It's not a problem for my usecase, just to understand. Marked as solved)

            – MeanStreet
            Jan 3 at 13:27






          • 1





            @MeanStreet The difference is that wait() returns sets of futures. Given a future, you can extract its result with result(), which will either return the result or re-raise the exception if there was one. But you can also call exception() to test whether an exception was raised. This allows you to distinguish between a function that raised an exception and one that (for whatever reason) happened to return an Exception instance.

            – user4815162342
            Jan 3 at 14:21













          • Oh I got it, that's what you meant by "mixes exceptions with regular results". Thanks

            – MeanStreet
            Jan 3 at 14:33
















          1















          The problem is that sometimes f() raises an Exception and I'm not sure how to handle it.




          That will depend on what you want to do when an exception occurs. Remember that asyncio.gather() is convenience API that propagates exceptions by default to avoid blindly continuing in case of error. If you want to proceed in case of exception, you have other options:




          • Pass return_exceptions=True to gather - this will cause gather to return the exception objects along with other results. Convenient and easy to use, but mixes exceptions with regular results, which is a bit messy.


          • Use asyncio.wait() instead of asyncio.gather(). It returns a set of futures, for which you can test whether they completed by raising or by producing a result.


          • Wrap f() in your own function that catches the exception as you see fit. You considered and rejected this, but in some cases it's exactly the right approach.







          share|improve this answer
























          • Thanks a lot for your answer, that's what I was looking for. I ran a few tests don't see clearly the difference between the first 2 options, it seems to me that gather(*futures, return_exceptions=True) does the same thing than wait(futures) (except that wait does not respect order), in both cases exceptions are mixed with results right ? (It's not a problem for my usecase, just to understand. Marked as solved)

            – MeanStreet
            Jan 3 at 13:27






          • 1





            @MeanStreet The difference is that wait() returns sets of futures. Given a future, you can extract its result with result(), which will either return the result or re-raise the exception if there was one. But you can also call exception() to test whether an exception was raised. This allows you to distinguish between a function that raised an exception and one that (for whatever reason) happened to return an Exception instance.

            – user4815162342
            Jan 3 at 14:21













          • Oh I got it, that's what you meant by "mixes exceptions with regular results". Thanks

            – MeanStreet
            Jan 3 at 14:33














          1












          1








          1








          The problem is that sometimes f() raises an Exception and I'm not sure how to handle it.




          That will depend on what you want to do when an exception occurs. Remember that asyncio.gather() is convenience API that propagates exceptions by default to avoid blindly continuing in case of error. If you want to proceed in case of exception, you have other options:




          • Pass return_exceptions=True to gather - this will cause gather to return the exception objects along with other results. Convenient and easy to use, but mixes exceptions with regular results, which is a bit messy.


          • Use asyncio.wait() instead of asyncio.gather(). It returns a set of futures, for which you can test whether they completed by raising or by producing a result.


          • Wrap f() in your own function that catches the exception as you see fit. You considered and rejected this, but in some cases it's exactly the right approach.







          share|improve this answer














          The problem is that sometimes f() raises an Exception and I'm not sure how to handle it.




          That will depend on what you want to do when an exception occurs. Remember that asyncio.gather() is convenience API that propagates exceptions by default to avoid blindly continuing in case of error. If you want to proceed in case of exception, you have other options:




          • Pass return_exceptions=True to gather - this will cause gather to return the exception objects along with other results. Convenient and easy to use, but mixes exceptions with regular results, which is a bit messy.


          • Use asyncio.wait() instead of asyncio.gather(). It returns a set of futures, for which you can test whether they completed by raising or by producing a result.


          • Wrap f() in your own function that catches the exception as you see fit. You considered and rejected this, but in some cases it's exactly the right approach.








          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Jan 3 at 13:01









          user4815162342user4815162342

          64.1k594151




          64.1k594151













          • Thanks a lot for your answer, that's what I was looking for. I ran a few tests don't see clearly the difference between the first 2 options, it seems to me that gather(*futures, return_exceptions=True) does the same thing than wait(futures) (except that wait does not respect order), in both cases exceptions are mixed with results right ? (It's not a problem for my usecase, just to understand. Marked as solved)

            – MeanStreet
            Jan 3 at 13:27






          • 1





            @MeanStreet The difference is that wait() returns sets of futures. Given a future, you can extract its result with result(), which will either return the result or re-raise the exception if there was one. But you can also call exception() to test whether an exception was raised. This allows you to distinguish between a function that raised an exception and one that (for whatever reason) happened to return an Exception instance.

            – user4815162342
            Jan 3 at 14:21













          • Oh I got it, that's what you meant by "mixes exceptions with regular results". Thanks

            – MeanStreet
            Jan 3 at 14:33



















          • Thanks a lot for your answer, that's what I was looking for. I ran a few tests don't see clearly the difference between the first 2 options, it seems to me that gather(*futures, return_exceptions=True) does the same thing than wait(futures) (except that wait does not respect order), in both cases exceptions are mixed with results right ? (It's not a problem for my usecase, just to understand. Marked as solved)

            – MeanStreet
            Jan 3 at 13:27






          • 1





            @MeanStreet The difference is that wait() returns sets of futures. Given a future, you can extract its result with result(), which will either return the result or re-raise the exception if there was one. But you can also call exception() to test whether an exception was raised. This allows you to distinguish between a function that raised an exception and one that (for whatever reason) happened to return an Exception instance.

            – user4815162342
            Jan 3 at 14:21













          • Oh I got it, that's what you meant by "mixes exceptions with regular results". Thanks

            – MeanStreet
            Jan 3 at 14:33

















          Thanks a lot for your answer, that's what I was looking for. I ran a few tests don't see clearly the difference between the first 2 options, it seems to me that gather(*futures, return_exceptions=True) does the same thing than wait(futures) (except that wait does not respect order), in both cases exceptions are mixed with results right ? (It's not a problem for my usecase, just to understand. Marked as solved)

          – MeanStreet
          Jan 3 at 13:27





          Thanks a lot for your answer, that's what I was looking for. I ran a few tests don't see clearly the difference between the first 2 options, it seems to me that gather(*futures, return_exceptions=True) does the same thing than wait(futures) (except that wait does not respect order), in both cases exceptions are mixed with results right ? (It's not a problem for my usecase, just to understand. Marked as solved)

          – MeanStreet
          Jan 3 at 13:27




          1




          1





          @MeanStreet The difference is that wait() returns sets of futures. Given a future, you can extract its result with result(), which will either return the result or re-raise the exception if there was one. But you can also call exception() to test whether an exception was raised. This allows you to distinguish between a function that raised an exception and one that (for whatever reason) happened to return an Exception instance.

          – user4815162342
          Jan 3 at 14:21







          @MeanStreet The difference is that wait() returns sets of futures. Given a future, you can extract its result with result(), which will either return the result or re-raise the exception if there was one. But you can also call exception() to test whether an exception was raised. This allows you to distinguish between a function that raised an exception and one that (for whatever reason) happened to return an Exception instance.

          – user4815162342
          Jan 3 at 14:21















          Oh I got it, that's what you meant by "mixes exceptions with regular results". Thanks

          – MeanStreet
          Jan 3 at 14:33





          Oh I got it, that's what you meant by "mixes exceptions with regular results". Thanks

          – MeanStreet
          Jan 3 at 14:33




















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54022514%2fhow-to-handle-error-raise-asyncio-in-python3%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          Monofisismo

          Image
          Monofisismo (do grego: monos - "único, singular" e physis - "natureza") é o ponto de vista cristológico que defende que, depois da união do divino e do humano na encarnação histórica, Jesus Cristo, como encarnação do Filho ou Verbo ( Logos ) de Deus, teria apenas uma única "natureza", a divina, e não uma síntese de ambas. O monofisismo é contraposto pelo diofisismo (ou "diafisismo"), que defende que Jesus preservou em si as duas naturezas. Historicamente, o monofisismo se refere primordialmente à posição dos que (especialmente no Egito e, em menor grau, na Síria) rejeitaram as decisões do Concílio de Calcedônia em 451 (o quarto concílio ecumênico). Os membros mais moderados entre eles, porém, defendem a teologia "miafisita", que se tornou a oficial para as Igrejas Ortodoxas Orientais. Muitos ortodoxos orientais, porém, rejeitam essa classificação, mesmo como um termo genérico, mas ele é amplamente utilizado na literatura históri
          Read more

          Angular Downloading a file using contenturl with Basic Authentication

          Image
          0 I have a contentUrl when user clicks on that,file should be downloaded. If I try to browse the contentUrl in the browser it asks for Userid and password.I believe this is Basic Authentication.I dont want user to enter UserId and Password so I am passing it Authorization(Please see below code). Url looks some thing like this: http://XXX.XXX.XX.XXX:8080/flowable-rest/service/runtime/tasks/90875/attachments/90555/content let username : string = 'admin'; let pwd : string = 'test'; let headers = new Headers(); headers.set('Accept', 'text/json'); headers.append('Content-Type', 'application/json'); headers.set('Access-Control-Allow-Origin', '*'); headers.set('Access-Control-Allow-Methods','POST, GET, OPTIONS, PUT, DELETE'); head
          Read more

          Olmecas

          Image
          Monumento 1, uma das quatro cabeças colossais encontradas em La Venta, com altura aproximada de 3 metros Olmecas é a designação do povo e da civilização que estiveram na origem da antiga cultura pré-colombiana da Mesoamérica e que se desenvolveram nas regiões tropicais do centro-sul do atual México durante o pré-clássico, próximo de onde hoje estão localizados os estados mexicanos de Veracruz e Tabasco, no Istmo de Tehuantepec, numa zona designada área nuclear olmeca. A cultura olmeca floresceu nesta região aproximadamente entre 1500 e 400 a.C., [ 1 ] e crê-se que tenha sido a civilização-mãe de todas as civilizações mesoamericanas que se desenvolveram posteriormente. [ 2 ] No entanto, desconhece-se a sua exacta filiação étnica, ainda que existam numerosas hipóteses colocadas para tentar resolver esta questão. O etnónimo olmeca foi cunhado pelos arqueólogos do século XX, e não devem confundir-se com os muito posteriores olmecas-xicalancas que ocuparam vários locais do México
          Read more