Internal and External communication in Kafka












0














Flow:
On-Prem ------>Proxy--->Kafka



advertised.listeners=PLAINTEXT://proxyhostname:8080 - for external communication

listeners=PLAINTEXT://:9092 = for internal communication


•When we set both the property, the internal communication is not happening.(Replication issue and the consumer couldn’t connect via locally and we have to provide the proxy Ip for consumer communication)



•How we can effectively use both the property for internal and external communication?



•Any alternative idea to do the external and internal communication?






apache-kafka







share|improve this question









New contributor




satheesh S is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

























    0














    Flow:
    On-Prem ------>Proxy--->Kafka



    advertised.listeners=PLAINTEXT://proxyhostname:8080 - for external communication
    
listeners=PLAINTEXT://:9092 = for internal communication


    •When we set both the property, the internal communication is not happening.(Replication issue and the consumer couldn’t connect via locally and we have to provide the proxy Ip for consumer communication)



    •How we can effectively use both the property for internal and external communication?



    •Any alternative idea to do the external and internal communication?






    apache-kafka







    share|improve this question









    New contributor




    satheesh S is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.























      0












      0








      0







      Flow:
      On-Prem ------>Proxy--->Kafka



      advertised.listeners=PLAINTEXT://proxyhostname:8080 - for external communication
      
listeners=PLAINTEXT://:9092 = for internal communication


      •When we set both the property, the internal communication is not happening.(Replication issue and the consumer couldn’t connect via locally and we have to provide the proxy Ip for consumer communication)



      •How we can effectively use both the property for internal and external communication?



      •Any alternative idea to do the external and internal communication?






      apache-kafka







      share|improve this question









      New contributor




      satheesh S is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.











      Flow:
      On-Prem ------>Proxy--->Kafka



      advertised.listeners=PLAINTEXT://proxyhostname:8080 - for external communication
      
listeners=PLAINTEXT://:9092 = for internal communication


      •When we set both the property, the internal communication is not happening.(Replication issue and the consumer couldn’t connect via locally and we have to provide the proxy Ip for consumer communication)



      •How we can effectively use both the property for internal and external communication?



      •Any alternative idea to do the external and internal communication?






      apache-kafka




      apache-kafka






      share|improve this question









      New contributor




      satheesh S is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.











      share|improve this question









      New contributor




      satheesh S is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      share|improve this question




      share|improve this question








      edited 2 days ago









      Nishu Tayal

      11.5k73481




      11.5k73481






      New contributor




      satheesh S is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.









      asked 2 days ago









      satheesh S

      12




      12




      New contributor




      satheesh S is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.





      New contributor





      satheesh S is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.






      satheesh S is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.
























          1 Answer
          1






          active

          oldest

          votes


















          1














          It's very common to define multiple listeners and Kafka supports that very well.



          To define several listeners, you need to list all of them in advertised.listeners/listeners.



          If multiple listeners are going to use the same Security Protocol (PLAINTEXT), you also need to set listener.security.protocol.map to map custom names to Security Protocols. See broker configs in the Kafka Docs.



          For example:



          listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
          
advertised.listeners=INTERNAL://:9092,EXTERNAL://proxyhostname:8080
          
listeners=INTERNAL://:9092,EXTERNAL://:8080


          This maps 2 names EXTERNAL and INTERNAL (you can use any name you like, I just reused names from your question) to the PLAINTEXT security protocol. Then for each, it defines the port to listen to and the hostname to advertise in metadata responses.






          share|improve this answer





















          • We are using Cloudera here.
            – satheesh S
            yesterday










          • Thanks for the answer. We are using cloudera here. When we add the below property,Kafka server is showing the error. listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT advertised.listeners=INTERNAL://:9092,EXTERNAL://10.91.192.124:8080 2018-12-28 06:03:28,009 FATAL kafka.Kafka$: java.lang.IllegalArgumentException: Error creating broker listeners from 'INTERNAL://:9092,EXTERNAL://10.91.192.124:8080': No enum constant org.apache.kafka.common.protocol.SecurityProtocol.INTERNAL Please assist.
            – satheesh S
            yesterday










          • What is the broker version? It looks like your version does not support this feature. This was added in Kafka 0.10.2 (released ~2 years ago), see cwiki.apache.org/confluence/display/KAFKA/….
            – Mickael Maison
            yesterday










          • Thanks for the information.we are using 0.10.0+kafka2.1.2. Wiki link suggest that 0.10.2.0 and 0.11.0.0. Please suggest which one we can go ahead.
            – satheesh S
            yesterday












          • You need at least 0.10.2 to use this feature. Pre 0.10.2, the only option you have is to use different security protocols for the 2 listeners (or upgrade to a recent version as 0.10.0 is now very old!)
            – Mickael Maison
            yesterday











          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });






          satheesh S is a new contributor. Be nice, and check out our Code of Conduct.










          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53945143%2finternal-and-external-communication-in-kafka%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          1














          It's very common to define multiple listeners and Kafka supports that very well.



          To define several listeners, you need to list all of them in advertised.listeners/listeners.



          If multiple listeners are going to use the same Security Protocol (PLAINTEXT), you also need to set listener.security.protocol.map to map custom names to Security Protocols. See broker configs in the Kafka Docs.



          For example:



          listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
          
advertised.listeners=INTERNAL://:9092,EXTERNAL://proxyhostname:8080
          
listeners=INTERNAL://:9092,EXTERNAL://:8080


          This maps 2 names EXTERNAL and INTERNAL (you can use any name you like, I just reused names from your question) to the PLAINTEXT security protocol. Then for each, it defines the port to listen to and the hostname to advertise in metadata responses.






          share|improve this answer





















          • We are using Cloudera here.
            – satheesh S
            yesterday










          • Thanks for the answer. We are using cloudera here. When we add the below property,Kafka server is showing the error. listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT advertised.listeners=INTERNAL://:9092,EXTERNAL://10.91.192.124:8080 2018-12-28 06:03:28,009 FATAL kafka.Kafka$: java.lang.IllegalArgumentException: Error creating broker listeners from 'INTERNAL://:9092,EXTERNAL://10.91.192.124:8080': No enum constant org.apache.kafka.common.protocol.SecurityProtocol.INTERNAL Please assist.
            – satheesh S
            yesterday










          • What is the broker version? It looks like your version does not support this feature. This was added in Kafka 0.10.2 (released ~2 years ago), see cwiki.apache.org/confluence/display/KAFKA/….
            – Mickael Maison
            yesterday










          • Thanks for the information.we are using 0.10.0+kafka2.1.2. Wiki link suggest that 0.10.2.0 and 0.11.0.0. Please suggest which one we can go ahead.
            – satheesh S
            yesterday












          • You need at least 0.10.2 to use this feature. Pre 0.10.2, the only option you have is to use different security protocols for the 2 listeners (or upgrade to a recent version as 0.10.0 is now very old!)
            – Mickael Maison
            yesterday
















          1














          It's very common to define multiple listeners and Kafka supports that very well.



          To define several listeners, you need to list all of them in advertised.listeners/listeners.



          If multiple listeners are going to use the same Security Protocol (PLAINTEXT), you also need to set listener.security.protocol.map to map custom names to Security Protocols. See broker configs in the Kafka Docs.



          For example:



          listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
          
advertised.listeners=INTERNAL://:9092,EXTERNAL://proxyhostname:8080
          
listeners=INTERNAL://:9092,EXTERNAL://:8080


          This maps 2 names EXTERNAL and INTERNAL (you can use any name you like, I just reused names from your question) to the PLAINTEXT security protocol. Then for each, it defines the port to listen to and the hostname to advertise in metadata responses.






          share|improve this answer





















          • We are using Cloudera here.
            – satheesh S
            yesterday










          • Thanks for the answer. We are using cloudera here. When we add the below property,Kafka server is showing the error. listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT advertised.listeners=INTERNAL://:9092,EXTERNAL://10.91.192.124:8080 2018-12-28 06:03:28,009 FATAL kafka.Kafka$: java.lang.IllegalArgumentException: Error creating broker listeners from 'INTERNAL://:9092,EXTERNAL://10.91.192.124:8080': No enum constant org.apache.kafka.common.protocol.SecurityProtocol.INTERNAL Please assist.
            – satheesh S
            yesterday










          • What is the broker version? It looks like your version does not support this feature. This was added in Kafka 0.10.2 (released ~2 years ago), see cwiki.apache.org/confluence/display/KAFKA/….
            – Mickael Maison
            yesterday










          • Thanks for the information.we are using 0.10.0+kafka2.1.2. Wiki link suggest that 0.10.2.0 and 0.11.0.0. Please suggest which one we can go ahead.
            – satheesh S
            yesterday












          • You need at least 0.10.2 to use this feature. Pre 0.10.2, the only option you have is to use different security protocols for the 2 listeners (or upgrade to a recent version as 0.10.0 is now very old!)
            – Mickael Maison
            yesterday














          1












          1








          1






          It's very common to define multiple listeners and Kafka supports that very well.



          To define several listeners, you need to list all of them in advertised.listeners/listeners.



          If multiple listeners are going to use the same Security Protocol (PLAINTEXT), you also need to set listener.security.protocol.map to map custom names to Security Protocols. See broker configs in the Kafka Docs.



          For example:



          listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
          
advertised.listeners=INTERNAL://:9092,EXTERNAL://proxyhostname:8080
          
listeners=INTERNAL://:9092,EXTERNAL://:8080


          This maps 2 names EXTERNAL and INTERNAL (you can use any name you like, I just reused names from your question) to the PLAINTEXT security protocol. Then for each, it defines the port to listen to and the hostname to advertise in metadata responses.






          share|improve this answer












          It's very common to define multiple listeners and Kafka supports that very well.



          To define several listeners, you need to list all of them in advertised.listeners/listeners.



          If multiple listeners are going to use the same Security Protocol (PLAINTEXT), you also need to set listener.security.protocol.map to map custom names to Security Protocols. See broker configs in the Kafka Docs.



          For example:



          listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
          
advertised.listeners=INTERNAL://:9092,EXTERNAL://proxyhostname:8080
          
listeners=INTERNAL://:9092,EXTERNAL://:8080


          This maps 2 names EXTERNAL and INTERNAL (you can use any name you like, I just reused names from your question) to the PLAINTEXT security protocol. Then for each, it defines the port to listen to and the hostname to advertise in metadata responses.







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered 2 days ago









          Mickael Maison

          7,00932529




          7,00932529












          • We are using Cloudera here.
            – satheesh S
            yesterday










          • Thanks for the answer. We are using cloudera here. When we add the below property,Kafka server is showing the error. listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT advertised.listeners=INTERNAL://:9092,EXTERNAL://10.91.192.124:8080 2018-12-28 06:03:28,009 FATAL kafka.Kafka$: java.lang.IllegalArgumentException: Error creating broker listeners from 'INTERNAL://:9092,EXTERNAL://10.91.192.124:8080': No enum constant org.apache.kafka.common.protocol.SecurityProtocol.INTERNAL Please assist.
            – satheesh S
            yesterday










          • What is the broker version? It looks like your version does not support this feature. This was added in Kafka 0.10.2 (released ~2 years ago), see cwiki.apache.org/confluence/display/KAFKA/….
            – Mickael Maison
            yesterday










          • Thanks for the information.we are using 0.10.0+kafka2.1.2. Wiki link suggest that 0.10.2.0 and 0.11.0.0. Please suggest which one we can go ahead.
            – satheesh S
            yesterday












          • You need at least 0.10.2 to use this feature. Pre 0.10.2, the only option you have is to use different security protocols for the 2 listeners (or upgrade to a recent version as 0.10.0 is now very old!)
            – Mickael Maison
            yesterday


















          • We are using Cloudera here.
            – satheesh S
            yesterday










          • Thanks for the answer. We are using cloudera here. When we add the below property,Kafka server is showing the error. listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT advertised.listeners=INTERNAL://:9092,EXTERNAL://10.91.192.124:8080 2018-12-28 06:03:28,009 FATAL kafka.Kafka$: java.lang.IllegalArgumentException: Error creating broker listeners from 'INTERNAL://:9092,EXTERNAL://10.91.192.124:8080': No enum constant org.apache.kafka.common.protocol.SecurityProtocol.INTERNAL Please assist.
            – satheesh S
            yesterday










          • What is the broker version? It looks like your version does not support this feature. This was added in Kafka 0.10.2 (released ~2 years ago), see cwiki.apache.org/confluence/display/KAFKA/….
            – Mickael Maison
            yesterday










          • Thanks for the information.we are using 0.10.0+kafka2.1.2. Wiki link suggest that 0.10.2.0 and 0.11.0.0. Please suggest which one we can go ahead.
            – satheesh S
            yesterday












          • You need at least 0.10.2 to use this feature. Pre 0.10.2, the only option you have is to use different security protocols for the 2 listeners (or upgrade to a recent version as 0.10.0 is now very old!)
            – Mickael Maison
            yesterday
















          We are using Cloudera here.
          – satheesh S
          yesterday




          We are using Cloudera here.
          – satheesh S
          yesterday












          Thanks for the answer. We are using cloudera here. When we add the below property,Kafka server is showing the error. listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT advertised.listeners=INTERNAL://:9092,EXTERNAL://10.91.192.124:8080 2018-12-28 06:03:28,009 FATAL kafka.Kafka$: java.lang.IllegalArgumentException: Error creating broker listeners from 'INTERNAL://:9092,EXTERNAL://10.91.192.124:8080': No enum constant org.apache.kafka.common.protocol.SecurityProtocol.INTERNAL Please assist.
          – satheesh S
          yesterday




          Thanks for the answer. We are using cloudera here. When we add the below property,Kafka server is showing the error. listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT advertised.listeners=INTERNAL://:9092,EXTERNAL://10.91.192.124:8080 2018-12-28 06:03:28,009 FATAL kafka.Kafka$: java.lang.IllegalArgumentException: Error creating broker listeners from 'INTERNAL://:9092,EXTERNAL://10.91.192.124:8080': No enum constant org.apache.kafka.common.protocol.SecurityProtocol.INTERNAL Please assist.
          – satheesh S
          yesterday












          What is the broker version? It looks like your version does not support this feature. This was added in Kafka 0.10.2 (released ~2 years ago), see cwiki.apache.org/confluence/display/KAFKA/….
          – Mickael Maison
          yesterday




          What is the broker version? It looks like your version does not support this feature. This was added in Kafka 0.10.2 (released ~2 years ago), see cwiki.apache.org/confluence/display/KAFKA/….
          – Mickael Maison
          yesterday












          Thanks for the information.we are using 0.10.0+kafka2.1.2. Wiki link suggest that 0.10.2.0 and 0.11.0.0. Please suggest which one we can go ahead.
          – satheesh S
          yesterday






          Thanks for the information.we are using 0.10.0+kafka2.1.2. Wiki link suggest that 0.10.2.0 and 0.11.0.0. Please suggest which one we can go ahead.
          – satheesh S
          yesterday














          You need at least 0.10.2 to use this feature. Pre 0.10.2, the only option you have is to use different security protocols for the 2 listeners (or upgrade to a recent version as 0.10.0 is now very old!)
          – Mickael Maison
          yesterday




          You need at least 0.10.2 to use this feature. Pre 0.10.2, the only option you have is to use different security protocols for the 2 listeners (or upgrade to a recent version as 0.10.0 is now very old!)
          – Mickael Maison
          yesterday










          satheesh S is a new contributor. Be nice, and check out our Code of Conduct.










          draft saved

          draft discarded


















          satheesh S is a new contributor. Be nice, and check out our Code of Conduct.













          satheesh S is a new contributor. Be nice, and check out our Code of Conduct.












          satheesh S is a new contributor. Be nice, and check out our Code of Conduct.
















          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.





          Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


          Please pay close attention to the following guidance:


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53945143%2finternal-and-external-communication-in-kafka%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          Monofisismo

          Image
          Monofisismo (do grego: monos - "único, singular" e physis - "natureza") é o ponto de vista cristológico que defende que, depois da união do divino e do humano na encarnação histórica, Jesus Cristo, como encarnação do Filho ou Verbo ( Logos ) de Deus, teria apenas uma única "natureza", a divina, e não uma síntese de ambas. O monofisismo é contraposto pelo diofisismo (ou "diafisismo"), que defende que Jesus preservou em si as duas naturezas. Historicamente, o monofisismo se refere primordialmente à posição dos que (especialmente no Egito e, em menor grau, na Síria) rejeitaram as decisões do Concílio de Calcedônia em 451 (o quarto concílio ecumênico). Os membros mais moderados entre eles, porém, defendem a teologia "miafisita", que se tornou a oficial para as Igrejas Ortodoxas Orientais. Muitos ortodoxos orientais, porém, rejeitam essa classificação, mesmo como um termo genérico, mas ele é amplamente utilizado na literatura históri...
          Read more

          Angular Downloading a file using contenturl with Basic Authentication

          Image
          0 I have a contentUrl when user clicks on that,file should be downloaded. If I try to browse the contentUrl in the browser it asks for Userid and password.I believe this is Basic Authentication.I dont want user to enter UserId and Password so I am passing it Authorization(Please see below code). Url looks some thing like this: http://XXX.XXX.XX.XXX:8080/flowable-rest/service/runtime/tasks/90875/attachments/90555/content let username : string = 'admin'; let pwd : string = 'test'; let headers = new Headers(); headers.set('Accept', 'text/json'); headers.append('Content-Type', 'application/json'); headers.set('Access-Control-Allow-Origin', '*'); headers.set('Access-Control-Allow-Methods','POST, GET, OPTIONS, PUT, DELETE'); head...
          Read more

          Olmecas

          Image
          Monumento 1, uma das quatro cabeças colossais encontradas em La Venta, com altura aproximada de 3 metros Olmecas é a designação do povo e da civilização que estiveram na origem da antiga cultura pré-colombiana da Mesoamérica e que se desenvolveram nas regiões tropicais do centro-sul do atual México durante o pré-clássico, próximo de onde hoje estão localizados os estados mexicanos de Veracruz e Tabasco, no Istmo de Tehuantepec, numa zona designada área nuclear olmeca. A cultura olmeca floresceu nesta região aproximadamente entre 1500 e 400 a.C., [ 1 ] e crê-se que tenha sido a civilização-mãe de todas as civilizações mesoamericanas que se desenvolveram posteriormente. [ 2 ] No entanto, desconhece-se a sua exacta filiação étnica, ainda que existam numerosas hipóteses colocadas para tentar resolver esta questão. O etnónimo olmeca foi cunhado pelos arqueólogos do século XX, e não devem confundir-se com os muito posteriores olmecas-xicalancas que ocuparam vários locais do México...
          Read more