TimescaleDB and PipelineDB Integration
(I was going to post this on the DBA SE, but there's no timescale tag.)
I'm using TimescaleDB to store a real-time stream of about 500-1000 inserts per second, and need to pass this data to a client application as soon as it comes in.
I don't necessarily want to process the raw millisecond data (I may), but I definitely need the client to receive 1, 5, 10, 30 second (etc.) aggregations of this live data, as each time bucket/slot is ready. So, every second, every 5 seconds, every 10 seconds, etc. (the values would be things like max, min, etc. for each time slot).
I was going to use Postgres' triggers to aggregate the data and put it in a new table in a waterfall type manner, and use listen/notify to tell the client when each time chunk is "ready", but then I came across PipelineDB.
However, it's not clear to me exactly how to set that architecture up. Do I want PipelineDB to get the initial data stream, and then pass it to the Timescale hyper table? Do I want Timescale to pass the data to Pipeline? I assume Pipeline would be the point of contact for the client.
What would the most performant general architecture be; and how would I pass data between Timescale and Pipeline?
streaming aggregation real-time-data pipelinedb timescaledb
asked Jan 1 at 21:22
The AI ArchitectThe AI Architect
8892918
add a comment |
(I was going to post this on the DBA SE, but there's no timescale tag.)
I'm using TimescaleDB to store a real-time stream of about 500-1000 inserts per second, and need to pass this data to a client application as soon as it comes in.
I don't necessarily want to process the raw millisecond data (I may), but I definitely need the client to receive 1, 5, 10, 30 second (etc.) aggregations of this live data, as each time bucket/slot is ready. So, every second, every 5 seconds, every 10 seconds, etc. (the values would be things like max, min, etc. for each time slot).
I was going to use Postgres' triggers to aggregate the data and put it in a new table in a waterfall type manner, and use listen/notify to tell the client when each time chunk is "ready", but then I came across PipelineDB.
However, it's not clear to me exactly how to set that architecture up. Do I want PipelineDB to get the initial data stream, and then pass it to the Timescale hyper table? Do I want Timescale to pass the data to Pipeline? I assume Pipeline would be the point of contact for the client.
What would the most performant general architecture be; and how would I pass data between Timescale and Pipeline?
streaming aggregation real-time-data pipelinedb timescaledb
asked Jan 1 at 21:22
The AI ArchitectThe AI Architect
8892918
add a comment |
(I was going to post this on the DBA SE, but there's no timescale tag.)
I'm using TimescaleDB to store a real-time stream of about 500-1000 inserts per second, and need to pass this data to a client application as soon as it comes in.
I don't necessarily want to process the raw millisecond data (I may), but I definitely need the client to receive 1, 5, 10, 30 second (etc.) aggregations of this live data, as each time bucket/slot is ready. So, every second, every 5 seconds, every 10 seconds, etc. (the values would be things like max, min, etc. for each time slot).
I was going to use Postgres' triggers to aggregate the data and put it in a new table in a waterfall type manner, and use listen/notify to tell the client when each time chunk is "ready", but then I came across PipelineDB.
However, it's not clear to me exactly how to set that architecture up. Do I want PipelineDB to get the initial data stream, and then pass it to the Timescale hyper table? Do I want Timescale to pass the data to Pipeline? I assume Pipeline would be the point of contact for the client.
What would the most performant general architecture be; and how would I pass data between Timescale and Pipeline?
streaming aggregation real-time-data pipelinedb timescaledb
asked Jan 1 at 21:22
The AI ArchitectThe AI Architect
8892918
(I was going to post this on the DBA SE, but there's no timescale tag.)
I'm using TimescaleDB to store a real-time stream of about 500-1000 inserts per second, and need to pass this data to a client application as soon as it comes in.
I don't necessarily want to process the raw millisecond data (I may), but I definitely need the client to receive 1, 5, 10, 30 second (etc.) aggregations of this live data, as each time bucket/slot is ready. So, every second, every 5 seconds, every 10 seconds, etc. (the values would be things like max, min, etc. for each time slot).
I was going to use Postgres' triggers to aggregate the data and put it in a new table in a waterfall type manner, and use listen/notify to tell the client when each time chunk is "ready", but then I came across PipelineDB.
However, it's not clear to me exactly how to set that architecture up. Do I want PipelineDB to get the initial data stream, and then pass it to the Timescale hyper table? Do I want Timescale to pass the data to Pipeline? I assume Pipeline would be the point of contact for the client.
What would the most performant general architecture be; and how would I pass data between Timescale and Pipeline?
streaming aggregation real-time-data pipelinedb timescaledb
streaming aggregation real-time-data pipelinedb timescaledb
asked Jan 1 at 21:22
The AI ArchitectThe AI Architect
8892918
asked Jan 1 at 21:22
The AI ArchitectThe AI Architect
8892918
asked Jan 1 at 21:22
The AI ArchitectThe AI Architect
8892918
asked Jan 1 at 21:22
The AI ArchitectThe AI Architect
8892918
asked Jan 1 at 21:22
The AI ArchitectThe AI Architect
8892918
8892918
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Currently there is no native integration between PipelineDB and TimescaleDB but there likely will be in the not-so-distant future. Performance wise, the best option is currently to simply bifurcate writes to both TimescaleDB and PipelineDB.
answered Jan 3 at 19:10
Derek NelsonDerek Nelson
21315
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53999037%2ftimescaledb-and-pipelinedb-integration%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Currently there is no native integration between PipelineDB and TimescaleDB but there likely will be in the not-so-distant future. Performance wise, the best option is currently to simply bifurcate writes to both TimescaleDB and PipelineDB.
answered Jan 3 at 19:10
Derek NelsonDerek Nelson
21315
add a comment |
Currently there is no native integration between PipelineDB and TimescaleDB but there likely will be in the not-so-distant future. Performance wise, the best option is currently to simply bifurcate writes to both TimescaleDB and PipelineDB.
answered Jan 3 at 19:10
Derek NelsonDerek Nelson
21315
add a comment |
Currently there is no native integration between PipelineDB and TimescaleDB but there likely will be in the not-so-distant future. Performance wise, the best option is currently to simply bifurcate writes to both TimescaleDB and PipelineDB.
answered Jan 3 at 19:10
Derek NelsonDerek Nelson
21315
Currently there is no native integration between PipelineDB and TimescaleDB but there likely will be in the not-so-distant future. Performance wise, the best option is currently to simply bifurcate writes to both TimescaleDB and PipelineDB.
answered Jan 3 at 19:10
Derek NelsonDerek Nelson
21315
answered Jan 3 at 19:10
Derek NelsonDerek Nelson
21315
answered Jan 3 at 19:10
Derek NelsonDerek Nelson
21315
answered Jan 3 at 19:10
Derek NelsonDerek Nelson
21315
21315
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53999037%2ftimescaledb-and-pipelinedb-integration%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
