How to ensure referential integrity in Spark structured streaming application?
Till now we have a batch processing application, where we asked the source to inform us when to start our application based on their input signal called "trigger". So once we receive it, our application start consuming the data till that point by considering as a single batch.
But now we want to handle this in our application without dependent on Source input trigger message. In the new Spark streaming application, we want to consume and process the same Kafka topics.
apache-spark apache-kafka spark-streaming referential-integrity
edited Dec 28 '18 at 2:19
cricket_007
79.5k1142109
asked Dec 28 '18 at 0:02
jeevan kishore
8312
|
show 1 more comment
Till now we have a batch processing application, where we asked the source to inform us when to start our application based on their input signal called "trigger". So once we receive it, our application start consuming the data till that point by considering as a single batch.
But now we want to handle this in our application without dependent on Source input trigger message. In the new Spark streaming application, we want to consume and process the same Kafka topics.
apache-spark apache-kafka spark-streaming referential-integrity
edited Dec 28 '18 at 2:19
cricket_007
79.5k1142109
asked Dec 28 '18 at 0:02
jeevan kishore
8312
What exactly is the problem you're having?
– cricket_007
Dec 28 '18 at 2:21
How we can be independent of trigger signal?
– jeevan kishore
Dec 28 '18 at 7:29
What determines when the trigger is fired, is it an end-of-day process or something?
– J Smith
Dec 28 '18 at 14:50
Reading of Kafka events isn't "triggered"... You have a continuous stream of events, and Spark essentially has an "infinite while loop" polling this data... If you really want a "trigger", then you actually still need a batch application
– cricket_007
Dec 28 '18 at 17:33
I'm not sure that's what's being implied - @jeevankishore could you clarify here?
– J Smith
Jan 2 at 23:05
|
show 1 more comment
Till now we have a batch processing application, where we asked the source to inform us when to start our application based on their input signal called "trigger". So once we receive it, our application start consuming the data till that point by considering as a single batch.
But now we want to handle this in our application without dependent on Source input trigger message. In the new Spark streaming application, we want to consume and process the same Kafka topics.
apache-spark apache-kafka spark-streaming referential-integrity
edited Dec 28 '18 at 2:19
cricket_007
79.5k1142109
asked Dec 28 '18 at 0:02
jeevan kishore
8312
Till now we have a batch processing application, where we asked the source to inform us when to start our application based on their input signal called "trigger". So once we receive it, our application start consuming the data till that point by considering as a single batch.
But now we want to handle this in our application without dependent on Source input trigger message. In the new Spark streaming application, we want to consume and process the same Kafka topics.
apache-spark apache-kafka spark-streaming referential-integrity
apache-spark apache-kafka spark-streaming referential-integrity
edited Dec 28 '18 at 2:19
cricket_007
79.5k1142109
asked Dec 28 '18 at 0:02
jeevan kishore
8312
edited Dec 28 '18 at 2:19
cricket_007
79.5k1142109
asked Dec 28 '18 at 0:02
jeevan kishore
8312
edited Dec 28 '18 at 2:19
cricket_007
79.5k1142109
edited Dec 28 '18 at 2:19
cricket_007
79.5k1142109
edited Dec 28 '18 at 2:19
cricket_007
79.5k1142109
79.5k1142109
asked Dec 28 '18 at 0:02
jeevan kishore
8312
asked Dec 28 '18 at 0:02
jeevan kishore
8312
asked Dec 28 '18 at 0:02
jeevan kishore
8312
8312
What exactly is the problem you're having?
– cricket_007
Dec 28 '18 at 2:21
How we can be independent of trigger signal?
– jeevan kishore
Dec 28 '18 at 7:29
What determines when the trigger is fired, is it an end-of-day process or something?
– J Smith
Dec 28 '18 at 14:50
Reading of Kafka events isn't "triggered"... You have a continuous stream of events, and Spark essentially has an "infinite while loop" polling this data... If you really want a "trigger", then you actually still need a batch application
– cricket_007
Dec 28 '18 at 17:33
I'm not sure that's what's being implied - @jeevankishore could you clarify here?
– J Smith
Jan 2 at 23:05
|
show 1 more comment
What exactly is the problem you're having?
– cricket_007
Dec 28 '18 at 2:21
How we can be independent of trigger signal?
– jeevan kishore
Dec 28 '18 at 7:29
What determines when the trigger is fired, is it an end-of-day process or something?
– J Smith
Dec 28 '18 at 14:50
Reading of Kafka events isn't "triggered"... You have a continuous stream of events, and Spark essentially has an "infinite while loop" polling this data... If you really want a "trigger", then you actually still need a batch application
– cricket_007
Dec 28 '18 at 17:33
I'm not sure that's what's being implied - @jeevankishore could you clarify here?
– J Smith
Jan 2 at 23:05
What exactly is the problem you're having?
– cricket_007
Dec 28 '18 at 2:21
What exactly is the problem you're having?
– cricket_007
Dec 28 '18 at 2:21
How we can be independent of trigger signal?
– jeevan kishore
Dec 28 '18 at 7:29
How we can be independent of trigger signal?
– jeevan kishore
Dec 28 '18 at 7:29
What determines when the trigger is fired, is it an end-of-day process or something?
– J Smith
Dec 28 '18 at 14:50
What determines when the trigger is fired, is it an end-of-day process or something?
– J Smith
Dec 28 '18 at 14:50
Reading of Kafka events isn't "triggered"... You have a continuous stream of events, and Spark essentially has an "infinite while loop" polling this data... If you really want a "trigger", then you actually still need a batch application
– cricket_007
Dec 28 '18 at 17:33
Reading of Kafka events isn't "triggered"... You have a continuous stream of events, and Spark essentially has an "infinite while loop" polling this data... If you really want a "trigger", then you actually still need a batch application
– cricket_007
Dec 28 '18 at 17:33
I'm not sure that's what's being implied - @jeevankishore could you clarify here?
– J Smith
Jan 2 at 23:05
I'm not sure that's what's being implied - @jeevankishore could you clarify here?
– J Smith
Jan 2 at 23:05
|
show 1 more comment
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53952207%2fhow-to-ensure-referential-integrity-in-spark-structured-streaming-application%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53952207%2fhow-to-ensure-referential-integrity-in-spark-structured-streaming-application%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What exactly is the problem you're having?
– cricket_007
Dec 28 '18 at 2:21
How we can be independent of trigger signal?
– jeevan kishore
Dec 28 '18 at 7:29
What determines when the trigger is fired, is it an end-of-day process or something?
– J Smith
Dec 28 '18 at 14:50
Reading of Kafka events isn't "triggered"... You have a continuous stream of events, and Spark essentially has an "infinite while loop" polling this data... If you really want a "trigger", then you actually still need a batch application
– cricket_007
Dec 28 '18 at 17:33
I'm not sure that's what's being implied - @jeevankishore could you clarify here?
– J Smith
Jan 2 at 23:05