Spring Kafka Listener on Http Request
I am using Spring and Kafka, I make an HTTP POST request as shown below and send some info to another service via a Kafka topic.
@RequestMapping(method = RequestMethod.POST, value = "/portfolio")
public void getPortfolio(
Authentication auth,
@RequestBody User user
) {
//Data Transfer Object
UserDTO dto = user.toDTO();
dto.setId(((AuthenticatedUser) auth.getPrincipal()).getId());
//Sending message to Kafka topic
sender.sendPortfolioRequest(dto);
}
I then want to listen for the response on a different topic and return data in the HTTP response but am stuck here. I am able to listen to for the response using the below listener method but don't know how to put the two together.
@KafkaListener(
topics = Topics.PORTFOLIO_RESULT,
containerFactory = "portfolioKafkaListenerContainerFactory"
)
public void portfolioListener(UserPortfolioDTO portfolio) {
System.out.println("Recieved Portfolio: " + portfolio.toString());
}
P.S. I am new to using HTTP requests and don't know if this is the correct way to do what I'm trying to achieve or if I should be creating a new resource with the POST and redirecting to that or something else.
java spring apache-kafka spring-kafka
asked Jan 3 at 15:39
Rory HarrisonRory Harrison
587
add a comment |
I am using Spring and Kafka, I make an HTTP POST request as shown below and send some info to another service via a Kafka topic.
@RequestMapping(method = RequestMethod.POST, value = "/portfolio")
public void getPortfolio(
Authentication auth,
@RequestBody User user
) {
//Data Transfer Object
UserDTO dto = user.toDTO();
dto.setId(((AuthenticatedUser) auth.getPrincipal()).getId());
//Sending message to Kafka topic
sender.sendPortfolioRequest(dto);
}
I then want to listen for the response on a different topic and return data in the HTTP response but am stuck here. I am able to listen to for the response using the below listener method but don't know how to put the two together.
@KafkaListener(
topics = Topics.PORTFOLIO_RESULT,
containerFactory = "portfolioKafkaListenerContainerFactory"
)
public void portfolioListener(UserPortfolioDTO portfolio) {
System.out.println("Recieved Portfolio: " + portfolio.toString());
}
P.S. I am new to using HTTP requests and don't know if this is the correct way to do what I'm trying to achieve or if I should be creating a new resource with the POST and redirecting to that or something else.
java spring apache-kafka spring-kafka
asked Jan 3 at 15:39
Rory HarrisonRory Harrison
587
add a comment |
I am using Spring and Kafka, I make an HTTP POST request as shown below and send some info to another service via a Kafka topic.
@RequestMapping(method = RequestMethod.POST, value = "/portfolio")
public void getPortfolio(
Authentication auth,
@RequestBody User user
) {
//Data Transfer Object
UserDTO dto = user.toDTO();
dto.setId(((AuthenticatedUser) auth.getPrincipal()).getId());
//Sending message to Kafka topic
sender.sendPortfolioRequest(dto);
}
I then want to listen for the response on a different topic and return data in the HTTP response but am stuck here. I am able to listen to for the response using the below listener method but don't know how to put the two together.
@KafkaListener(
topics = Topics.PORTFOLIO_RESULT,
containerFactory = "portfolioKafkaListenerContainerFactory"
)
public void portfolioListener(UserPortfolioDTO portfolio) {
System.out.println("Recieved Portfolio: " + portfolio.toString());
}
P.S. I am new to using HTTP requests and don't know if this is the correct way to do what I'm trying to achieve or if I should be creating a new resource with the POST and redirecting to that or something else.
java spring apache-kafka spring-kafka
asked Jan 3 at 15:39
Rory HarrisonRory Harrison
587
I am using Spring and Kafka, I make an HTTP POST request as shown below and send some info to another service via a Kafka topic.
@RequestMapping(method = RequestMethod.POST, value = "/portfolio")
public void getPortfolio(
Authentication auth,
@RequestBody User user
) {
//Data Transfer Object
UserDTO dto = user.toDTO();
dto.setId(((AuthenticatedUser) auth.getPrincipal()).getId());
//Sending message to Kafka topic
sender.sendPortfolioRequest(dto);
}
I then want to listen for the response on a different topic and return data in the HTTP response but am stuck here. I am able to listen to for the response using the below listener method but don't know how to put the two together.
@KafkaListener(
topics = Topics.PORTFOLIO_RESULT,
containerFactory = "portfolioKafkaListenerContainerFactory"
)
public void portfolioListener(UserPortfolioDTO portfolio) {
System.out.println("Recieved Portfolio: " + portfolio.toString());
}
P.S. I am new to using HTTP requests and don't know if this is the correct way to do what I'm trying to achieve or if I should be creating a new resource with the POST and redirecting to that or something else.
java spring apache-kafka spring-kafka
java spring apache-kafka spring-kafka
asked Jan 3 at 15:39
Rory HarrisonRory Harrison
587
asked Jan 3 at 15:39
Rory HarrisonRory Harrison
587
asked Jan 3 at 15:39
Rory HarrisonRory Harrison
587
asked Jan 3 at 15:39
Rory HarrisonRory Harrison
587
asked Jan 3 at 15:39
Rory HarrisonRory Harrison
587
587
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
This can't be done with the @KafkaListener since it is started separately and fully works in its own thread(s). Meanwhile you expect a reply in the HTTP request thread.
Only the solution possible for you here is a ConsumerFactory and manual usage of the Apache Kafka Consumer. So, you send, you obtain a Consumer instance from the factory, call its poll() being blocked until a result, build a response for HTTP and close Consumer.
answered Jan 3 at 15:48
Artem BilanArtem Bilan
67.7k84973
2
You can use aReplyingKafkaTemplate.sendAndReceive()method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a@KafkaListenerwith a return value. You can either block for the reply or add a listener to the future to reply asynchronously.
– Gary Russell
Jan 3 at 16:09
add a comment |
According to what I'm understanding, I suggest you to enable asych http request to be able to link your process.
Creating Asynchronous Methods with springboot
This option will allow you to process sendPortfolioRequest and free the http request (otherwise you will have an http request timeout from client side).
And what you try to do looks like an anti-pattern : you want to do link a sychron http request (because your http client is waiting a response from the server) and an asynchron messaging system (Kafka).
To be able to do what you want, I suggest you to change your http endpoint and add a websocket to be in the best practice.
Please see Using WebSocket to build an interactive web application
answered Jan 3 at 16:06
bdzzaidbdzzaid
913
1
You can use aReplyingKafkaTemplate.sendAndReceive()method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a@KafkaListenerwith a return value. You can either block for the reply or add a listener to the future to reply asynchronously.
– Gary Russell
Jan 3 at 16:12
add a comment |
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54025437%2fspring-kafka-listener-on-http-request%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
This can't be done with the @KafkaListener since it is started separately and fully works in its own thread(s). Meanwhile you expect a reply in the HTTP request thread.
Only the solution possible for you here is a ConsumerFactory and manual usage of the Apache Kafka Consumer. So, you send, you obtain a Consumer instance from the factory, call its poll() being blocked until a result, build a response for HTTP and close Consumer.
answered Jan 3 at 15:48
Artem BilanArtem Bilan
67.7k84973
2
You can use aReplyingKafkaTemplate.sendAndReceive()method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a@KafkaListenerwith a return value. You can either block for the reply or add a listener to the future to reply asynchronously.
– Gary Russell
Jan 3 at 16:09
add a comment |
This can't be done with the @KafkaListener since it is started separately and fully works in its own thread(s). Meanwhile you expect a reply in the HTTP request thread.
Only the solution possible for you here is a ConsumerFactory and manual usage of the Apache Kafka Consumer. So, you send, you obtain a Consumer instance from the factory, call its poll() being blocked until a result, build a response for HTTP and close Consumer.
answered Jan 3 at 15:48
Artem BilanArtem Bilan
67.7k84973
2
You can use aReplyingKafkaTemplate.sendAndReceive()method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a@KafkaListenerwith a return value. You can either block for the reply or add a listener to the future to reply asynchronously.
– Gary Russell
Jan 3 at 16:09
add a comment |
This can't be done with the @KafkaListener since it is started separately and fully works in its own thread(s). Meanwhile you expect a reply in the HTTP request thread.
Only the solution possible for you here is a ConsumerFactory and manual usage of the Apache Kafka Consumer. So, you send, you obtain a Consumer instance from the factory, call its poll() being blocked until a result, build a response for HTTP and close Consumer.
answered Jan 3 at 15:48
Artem BilanArtem Bilan
67.7k84973
This can't be done with the @KafkaListener since it is started separately and fully works in its own thread(s). Meanwhile you expect a reply in the HTTP request thread.
Only the solution possible for you here is a ConsumerFactory and manual usage of the Apache Kafka Consumer. So, you send, you obtain a Consumer instance from the factory, call its poll() being blocked until a result, build a response for HTTP and close Consumer.
answered Jan 3 at 15:48
Artem BilanArtem Bilan
67.7k84973
answered Jan 3 at 15:48
Artem BilanArtem Bilan
67.7k84973
answered Jan 3 at 15:48
Artem BilanArtem Bilan
67.7k84973
answered Jan 3 at 15:48
Artem BilanArtem Bilan
67.7k84973
67.7k84973
2
You can use aReplyingKafkaTemplate.sendAndReceive()method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a@KafkaListenerwith a return value. You can either block for the reply or add a listener to the future to reply asynchronously.
– Gary Russell
Jan 3 at 16:09
add a comment |
2
You can use aReplyingKafkaTemplate.sendAndReceive()method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a@KafkaListenerwith a return value. You can either block for the reply or add a listener to the future to reply asynchronously.
– Gary Russell
Jan 3 at 16:09
2
2
You can use a
ReplyingKafkaTemplate.sendAndReceive() method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a @KafkaListener with a return value. You can either block for the reply or add a listener to the future to reply asynchronously.– Gary Russell
Jan 3 at 16:09
You can use a
ReplyingKafkaTemplate.sendAndReceive() method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a @KafkaListener with a return value. You can either block for the reply or add a listener to the future to reply asynchronously.– Gary Russell
Jan 3 at 16:09
add a comment |
According to what I'm understanding, I suggest you to enable asych http request to be able to link your process.
Creating Asynchronous Methods with springboot
This option will allow you to process sendPortfolioRequest and free the http request (otherwise you will have an http request timeout from client side).
And what you try to do looks like an anti-pattern : you want to do link a sychron http request (because your http client is waiting a response from the server) and an asynchron messaging system (Kafka).
To be able to do what you want, I suggest you to change your http endpoint and add a websocket to be in the best practice.
Please see Using WebSocket to build an interactive web application
answered Jan 3 at 16:06
bdzzaidbdzzaid
913
1
You can use aReplyingKafkaTemplate.sendAndReceive()method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a@KafkaListenerwith a return value. You can either block for the reply or add a listener to the future to reply asynchronously.
– Gary Russell
Jan 3 at 16:12
add a comment |
According to what I'm understanding, I suggest you to enable asych http request to be able to link your process.
Creating Asynchronous Methods with springboot
This option will allow you to process sendPortfolioRequest and free the http request (otherwise you will have an http request timeout from client side).
And what you try to do looks like an anti-pattern : you want to do link a sychron http request (because your http client is waiting a response from the server) and an asynchron messaging system (Kafka).
To be able to do what you want, I suggest you to change your http endpoint and add a websocket to be in the best practice.
Please see Using WebSocket to build an interactive web application
answered Jan 3 at 16:06
bdzzaidbdzzaid
913
1
You can use aReplyingKafkaTemplate.sendAndReceive()method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a@KafkaListenerwith a return value. You can either block for the reply or add a listener to the future to reply asynchronously.
– Gary Russell
Jan 3 at 16:12
add a comment |
According to what I'm understanding, I suggest you to enable asych http request to be able to link your process.
Creating Asynchronous Methods with springboot
This option will allow you to process sendPortfolioRequest and free the http request (otherwise you will have an http request timeout from client side).
And what you try to do looks like an anti-pattern : you want to do link a sychron http request (because your http client is waiting a response from the server) and an asynchron messaging system (Kafka).
To be able to do what you want, I suggest you to change your http endpoint and add a websocket to be in the best practice.
Please see Using WebSocket to build an interactive web application
answered Jan 3 at 16:06
bdzzaidbdzzaid
913
According to what I'm understanding, I suggest you to enable asych http request to be able to link your process.
Creating Asynchronous Methods with springboot
This option will allow you to process sendPortfolioRequest and free the http request (otherwise you will have an http request timeout from client side).
And what you try to do looks like an anti-pattern : you want to do link a sychron http request (because your http client is waiting a response from the server) and an asynchron messaging system (Kafka).
To be able to do what you want, I suggest you to change your http endpoint and add a websocket to be in the best practice.
Please see Using WebSocket to build an interactive web application
answered Jan 3 at 16:06
bdzzaidbdzzaid
913
answered Jan 3 at 16:06
bdzzaidbdzzaid
913
answered Jan 3 at 16:06
bdzzaidbdzzaid
913
answered Jan 3 at 16:06
bdzzaidbdzzaid
913
913
1
You can use aReplyingKafkaTemplate.sendAndReceive()method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a@KafkaListenerwith a return value. You can either block for the reply or add a listener to the future to reply asynchronously.
– Gary Russell
Jan 3 at 16:12
add a comment |
1
You can use aReplyingKafkaTemplate.sendAndReceive()method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a@KafkaListenerwith a return value. You can either block for the reply or add a listener to the future to reply asynchronously.
– Gary Russell
Jan 3 at 16:12
1
1
You can use a
ReplyingKafkaTemplate.sendAndReceive() method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a @KafkaListener with a return value. You can either block for the reply or add a listener to the future to reply asynchronously.– Gary Russell
Jan 3 at 16:12
You can use a
ReplyingKafkaTemplate.sendAndReceive() method for this use case; the responding system must set up the correlation id in the reply, which is done automatically if it is a @KafkaListener with a return value. You can either block for the reply or add a listener to the future to reply asynchronously.– Gary Russell
Jan 3 at 16:12
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54025437%2fspring-kafka-listener-on-http-request%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
