Apache Kafka Connect With Springboot

Multi tool use
I'm trying to find examples of kafka connect with springboot. It looks like there is no spring boot integration for kafka connect. Can some one point me in the right direction to be able to listen to changes on mysql db?
spring-boot apache-kafka apache-kafka-connect
add a comment |
I'm trying to find examples of kafka connect with springboot. It looks like there is no spring boot integration for kafka connect. Can some one point me in the right direction to be able to listen to changes on mysql db?
spring-boot apache-kafka apache-kafka-connect
add a comment |
I'm trying to find examples of kafka connect with springboot. It looks like there is no spring boot integration for kafka connect. Can some one point me in the right direction to be able to listen to changes on mysql db?
spring-boot apache-kafka apache-kafka-connect
I'm trying to find examples of kafka connect with springboot. It looks like there is no spring boot integration for kafka connect. Can some one point me in the right direction to be able to listen to changes on mysql db?
spring-boot apache-kafka apache-kafka-connect
spring-boot apache-kafka apache-kafka-connect
edited Dec 30 '18 at 5:38
cricket_007
81k1142110
81k1142110
asked Dec 30 '18 at 0:31
user3310115user3310115
250311
250311
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Kafka Connect doesn't really need Spring Boot because there is nothing for you to code for it, and it really works best when ran in distrbited-mode, as a cluster, not embedded within other applications. I suppose if you did want to do it, then you could copy relevent portions of the source code, but that of course isn't using Spring Boot, and you'd have to wire it all yourself
The framework itself consists of a few core Java dependencies that have already been written (Debezium or Confluent JDBC Connector, for your mysql example), and two config files. One for Kafka Connect to know the bootstrap servers, serializers, etc. and another for the actual MySQL connector.
So then how can we integrate with a java application. Can you point out to some examples
– user3310115
Dec 30 '18 at 10:29
So its something like download docker image, and change config and run right? And one more question connector need to be install on database to stream the changes? Or its just a config changes?
– Deadpool
Dec 30 '18 at 15:36
@user331 I suggest searching for "Debezium Embedded" if you want to do this. As I said, out of the box, Kafka Connect isn't exactly designed to be ran as part of another application
– cricket_007
Dec 31 '18 at 2:03
@Deadpool Docker is just one option, not required, but yes, just configs. And depends on the database, like Mongo/MySql/Postgres use the replication oplog. Couchbase has its own CDC protocols, though, for example. These act over the network, though, not running on the database server itself
– cricket_007
Dec 31 '18 at 2:06
i'm using postgres database, so it will use the replication oplog, but i have a question some of the organizations does not allow access to log file, then any other options? @cricket_007
– Deadpool
Dec 31 '18 at 2:09
|
show 2 more comments
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53974402%2fapache-kafka-connect-with-springboot%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Kafka Connect doesn't really need Spring Boot because there is nothing for you to code for it, and it really works best when ran in distrbited-mode, as a cluster, not embedded within other applications. I suppose if you did want to do it, then you could copy relevent portions of the source code, but that of course isn't using Spring Boot, and you'd have to wire it all yourself
The framework itself consists of a few core Java dependencies that have already been written (Debezium or Confluent JDBC Connector, for your mysql example), and two config files. One for Kafka Connect to know the bootstrap servers, serializers, etc. and another for the actual MySQL connector.
So then how can we integrate with a java application. Can you point out to some examples
– user3310115
Dec 30 '18 at 10:29
So its something like download docker image, and change config and run right? And one more question connector need to be install on database to stream the changes? Or its just a config changes?
– Deadpool
Dec 30 '18 at 15:36
@user331 I suggest searching for "Debezium Embedded" if you want to do this. As I said, out of the box, Kafka Connect isn't exactly designed to be ran as part of another application
– cricket_007
Dec 31 '18 at 2:03
@Deadpool Docker is just one option, not required, but yes, just configs. And depends on the database, like Mongo/MySql/Postgres use the replication oplog. Couchbase has its own CDC protocols, though, for example. These act over the network, though, not running on the database server itself
– cricket_007
Dec 31 '18 at 2:06
i'm using postgres database, so it will use the replication oplog, but i have a question some of the organizations does not allow access to log file, then any other options? @cricket_007
– Deadpool
Dec 31 '18 at 2:09
|
show 2 more comments
Kafka Connect doesn't really need Spring Boot because there is nothing for you to code for it, and it really works best when ran in distrbited-mode, as a cluster, not embedded within other applications. I suppose if you did want to do it, then you could copy relevent portions of the source code, but that of course isn't using Spring Boot, and you'd have to wire it all yourself
The framework itself consists of a few core Java dependencies that have already been written (Debezium or Confluent JDBC Connector, for your mysql example), and two config files. One for Kafka Connect to know the bootstrap servers, serializers, etc. and another for the actual MySQL connector.
So then how can we integrate with a java application. Can you point out to some examples
– user3310115
Dec 30 '18 at 10:29
So its something like download docker image, and change config and run right? And one more question connector need to be install on database to stream the changes? Or its just a config changes?
– Deadpool
Dec 30 '18 at 15:36
@user331 I suggest searching for "Debezium Embedded" if you want to do this. As I said, out of the box, Kafka Connect isn't exactly designed to be ran as part of another application
– cricket_007
Dec 31 '18 at 2:03
@Deadpool Docker is just one option, not required, but yes, just configs. And depends on the database, like Mongo/MySql/Postgres use the replication oplog. Couchbase has its own CDC protocols, though, for example. These act over the network, though, not running on the database server itself
– cricket_007
Dec 31 '18 at 2:06
i'm using postgres database, so it will use the replication oplog, but i have a question some of the organizations does not allow access to log file, then any other options? @cricket_007
– Deadpool
Dec 31 '18 at 2:09
|
show 2 more comments
Kafka Connect doesn't really need Spring Boot because there is nothing for you to code for it, and it really works best when ran in distrbited-mode, as a cluster, not embedded within other applications. I suppose if you did want to do it, then you could copy relevent portions of the source code, but that of course isn't using Spring Boot, and you'd have to wire it all yourself
The framework itself consists of a few core Java dependencies that have already been written (Debezium or Confluent JDBC Connector, for your mysql example), and two config files. One for Kafka Connect to know the bootstrap servers, serializers, etc. and another for the actual MySQL connector.
Kafka Connect doesn't really need Spring Boot because there is nothing for you to code for it, and it really works best when ran in distrbited-mode, as a cluster, not embedded within other applications. I suppose if you did want to do it, then you could copy relevent portions of the source code, but that of course isn't using Spring Boot, and you'd have to wire it all yourself
The framework itself consists of a few core Java dependencies that have already been written (Debezium or Confluent JDBC Connector, for your mysql example), and two config files. One for Kafka Connect to know the bootstrap servers, serializers, etc. and another for the actual MySQL connector.
edited Jan 5 at 23:17
answered Dec 30 '18 at 5:39
cricket_007cricket_007
81k1142110
81k1142110
So then how can we integrate with a java application. Can you point out to some examples
– user3310115
Dec 30 '18 at 10:29
So its something like download docker image, and change config and run right? And one more question connector need to be install on database to stream the changes? Or its just a config changes?
– Deadpool
Dec 30 '18 at 15:36
@user331 I suggest searching for "Debezium Embedded" if you want to do this. As I said, out of the box, Kafka Connect isn't exactly designed to be ran as part of another application
– cricket_007
Dec 31 '18 at 2:03
@Deadpool Docker is just one option, not required, but yes, just configs. And depends on the database, like Mongo/MySql/Postgres use the replication oplog. Couchbase has its own CDC protocols, though, for example. These act over the network, though, not running on the database server itself
– cricket_007
Dec 31 '18 at 2:06
i'm using postgres database, so it will use the replication oplog, but i have a question some of the organizations does not allow access to log file, then any other options? @cricket_007
– Deadpool
Dec 31 '18 at 2:09
|
show 2 more comments
So then how can we integrate with a java application. Can you point out to some examples
– user3310115
Dec 30 '18 at 10:29
So its something like download docker image, and change config and run right? And one more question connector need to be install on database to stream the changes? Or its just a config changes?
– Deadpool
Dec 30 '18 at 15:36
@user331 I suggest searching for "Debezium Embedded" if you want to do this. As I said, out of the box, Kafka Connect isn't exactly designed to be ran as part of another application
– cricket_007
Dec 31 '18 at 2:03
@Deadpool Docker is just one option, not required, but yes, just configs. And depends on the database, like Mongo/MySql/Postgres use the replication oplog. Couchbase has its own CDC protocols, though, for example. These act over the network, though, not running on the database server itself
– cricket_007
Dec 31 '18 at 2:06
i'm using postgres database, so it will use the replication oplog, but i have a question some of the organizations does not allow access to log file, then any other options? @cricket_007
– Deadpool
Dec 31 '18 at 2:09
So then how can we integrate with a java application. Can you point out to some examples
– user3310115
Dec 30 '18 at 10:29
So then how can we integrate with a java application. Can you point out to some examples
– user3310115
Dec 30 '18 at 10:29
So its something like download docker image, and change config and run right? And one more question connector need to be install on database to stream the changes? Or its just a config changes?
– Deadpool
Dec 30 '18 at 15:36
So its something like download docker image, and change config and run right? And one more question connector need to be install on database to stream the changes? Or its just a config changes?
– Deadpool
Dec 30 '18 at 15:36
@user331 I suggest searching for "Debezium Embedded" if you want to do this. As I said, out of the box, Kafka Connect isn't exactly designed to be ran as part of another application
– cricket_007
Dec 31 '18 at 2:03
@user331 I suggest searching for "Debezium Embedded" if you want to do this. As I said, out of the box, Kafka Connect isn't exactly designed to be ran as part of another application
– cricket_007
Dec 31 '18 at 2:03
@Deadpool Docker is just one option, not required, but yes, just configs. And depends on the database, like Mongo/MySql/Postgres use the replication oplog. Couchbase has its own CDC protocols, though, for example. These act over the network, though, not running on the database server itself
– cricket_007
Dec 31 '18 at 2:06
@Deadpool Docker is just one option, not required, but yes, just configs. And depends on the database, like Mongo/MySql/Postgres use the replication oplog. Couchbase has its own CDC protocols, though, for example. These act over the network, though, not running on the database server itself
– cricket_007
Dec 31 '18 at 2:06
i'm using postgres database, so it will use the replication oplog, but i have a question some of the organizations does not allow access to log file, then any other options? @cricket_007
– Deadpool
Dec 31 '18 at 2:09
i'm using postgres database, so it will use the replication oplog, but i have a question some of the organizations does not allow access to log file, then any other options? @cricket_007
– Deadpool
Dec 31 '18 at 2:09
|
show 2 more comments
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53974402%2fapache-kafka-connect-with-springboot%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
hrPIERG2GUX240zmzUMR,P0KX gZ6FFyh F7o2 Ik9a,DporV,Z2Q,l,l,JDjm023ByO,1f AR2cNOm9LRI 8JbN,NPDab27o