How many users can MongooseIM handle on the PostgreSQL backend?

Multi tool use
I am working on MongooseIM server with PostgreSQL as backend. I need all the basic features like:
- One to One chat with presence.
- Group Chat (Muc light only)
- MAM storage and all other basic features.
I can scale MongooseIM with clustering (with Mnesia for storing sessions). But I feel I may face bottle neck from the backend side using PostgreSQL. What is the rough estimate of the number of users that can be handled by PostgreSQL without the must have need for shift on No-sql like Riak. Because I don't want to mess-up the backend from the scalability point of view.
Example - Suppose I need to handle >10 Million users. Can I use PostgreSQL for that. If yes, then do I need special techniques like sharding in Postgresql to continue to work with it or at some point I must have to be shifted to Riak or other Nosql. What do you recommend?
sql postgresql nosql mongoose-im
add a comment |
I am working on MongooseIM server with PostgreSQL as backend. I need all the basic features like:
- One to One chat with presence.
- Group Chat (Muc light only)
- MAM storage and all other basic features.
I can scale MongooseIM with clustering (with Mnesia for storing sessions). But I feel I may face bottle neck from the backend side using PostgreSQL. What is the rough estimate of the number of users that can be handled by PostgreSQL without the must have need for shift on No-sql like Riak. Because I don't want to mess-up the backend from the scalability point of view.
Example - Suppose I need to handle >10 Million users. Can I use PostgreSQL for that. If yes, then do I need special techniques like sharding in Postgresql to continue to work with it or at some point I must have to be shifted to Riak or other Nosql. What do you recommend?
sql postgresql nosql mongoose-im
add a comment |
I am working on MongooseIM server with PostgreSQL as backend. I need all the basic features like:
- One to One chat with presence.
- Group Chat (Muc light only)
- MAM storage and all other basic features.
I can scale MongooseIM with clustering (with Mnesia for storing sessions). But I feel I may face bottle neck from the backend side using PostgreSQL. What is the rough estimate of the number of users that can be handled by PostgreSQL without the must have need for shift on No-sql like Riak. Because I don't want to mess-up the backend from the scalability point of view.
Example - Suppose I need to handle >10 Million users. Can I use PostgreSQL for that. If yes, then do I need special techniques like sharding in Postgresql to continue to work with it or at some point I must have to be shifted to Riak or other Nosql. What do you recommend?
sql postgresql nosql mongoose-im
I am working on MongooseIM server with PostgreSQL as backend. I need all the basic features like:
- One to One chat with presence.
- Group Chat (Muc light only)
- MAM storage and all other basic features.
I can scale MongooseIM with clustering (with Mnesia for storing sessions). But I feel I may face bottle neck from the backend side using PostgreSQL. What is the rough estimate of the number of users that can be handled by PostgreSQL without the must have need for shift on No-sql like Riak. Because I don't want to mess-up the backend from the scalability point of view.
Example - Suppose I need to handle >10 Million users. Can I use PostgreSQL for that. If yes, then do I need special techniques like sharding in Postgresql to continue to work with it or at some point I must have to be shifted to Riak or other Nosql. What do you recommend?
sql postgresql nosql mongoose-im
sql postgresql nosql mongoose-im
edited Jan 2 at 10:33
Shubham1164
asked Dec 29 '18 at 18:03


Shubham1164Shubham1164
7117
7117
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
This is a complex matter. First of all, it depends a lot on the set of features you have enabled in MongooseIM, so any load tests done by others might not reflect your particular configuration and environment.
Moreover, to actually be able to monitor all the important parameters in a system reaching tens of thousands of users, you'll probably need client-side performance tracking (delivery acks, roundtrip time) - this is not standardised by XMPP XEPs though, so implementations vary.
This being said, Postgres will probably only allow you to scale into hundreds of thousands of online users. It might require table sharding and user sharding across MongooseIM clusters using XMPP federation to allow for communication between all users. DB nodes in this architecture are associated with MIM clusters, therefore do not share data across cluster boundaries. Setups along these lines are known to work in production.
If you have to handle millions of users, you most likely have to switch from PostgreSQL to a database which can itself be more easily scaled. MongooseIM supports Cassandra, which currently seems to be the best bet. Personally, I'd curious to find out how TiKV (let me know, please, if you try that one) fares in such setups. It pretends to be MySQL on the wire, so should also work with MongooseIM, but operationally behaves similarly to Cassandra: automated replication, cluster scalability, fault tolerance, etc.
Ultimately, though, you'll only know by testing yourself. Instagram shares some interesting stories about Postgres at scale which show the amount of maintenance required. Alas, load tests reaching millions of users are prohibitively expensive due to cloud costs, so such test results are not common.
Thank erszcz, that was very informative. It seems like I must not under-estimate the potencial of Postgresql and should not blindly pick up any nosql tech. Although nosql like Cassandra surely can be a wise option. But if Instagram can manage their work in Postgresql then I must take it serious too
– Shubham1164
Jan 2 at 14:23
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53972068%2fhow-many-users-can-mongooseim-handle-on-the-postgresql-backend%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
This is a complex matter. First of all, it depends a lot on the set of features you have enabled in MongooseIM, so any load tests done by others might not reflect your particular configuration and environment.
Moreover, to actually be able to monitor all the important parameters in a system reaching tens of thousands of users, you'll probably need client-side performance tracking (delivery acks, roundtrip time) - this is not standardised by XMPP XEPs though, so implementations vary.
This being said, Postgres will probably only allow you to scale into hundreds of thousands of online users. It might require table sharding and user sharding across MongooseIM clusters using XMPP federation to allow for communication between all users. DB nodes in this architecture are associated with MIM clusters, therefore do not share data across cluster boundaries. Setups along these lines are known to work in production.
If you have to handle millions of users, you most likely have to switch from PostgreSQL to a database which can itself be more easily scaled. MongooseIM supports Cassandra, which currently seems to be the best bet. Personally, I'd curious to find out how TiKV (let me know, please, if you try that one) fares in such setups. It pretends to be MySQL on the wire, so should also work with MongooseIM, but operationally behaves similarly to Cassandra: automated replication, cluster scalability, fault tolerance, etc.
Ultimately, though, you'll only know by testing yourself. Instagram shares some interesting stories about Postgres at scale which show the amount of maintenance required. Alas, load tests reaching millions of users are prohibitively expensive due to cloud costs, so such test results are not common.
Thank erszcz, that was very informative. It seems like I must not under-estimate the potencial of Postgresql and should not blindly pick up any nosql tech. Although nosql like Cassandra surely can be a wise option. But if Instagram can manage their work in Postgresql then I must take it serious too
– Shubham1164
Jan 2 at 14:23
add a comment |
This is a complex matter. First of all, it depends a lot on the set of features you have enabled in MongooseIM, so any load tests done by others might not reflect your particular configuration and environment.
Moreover, to actually be able to monitor all the important parameters in a system reaching tens of thousands of users, you'll probably need client-side performance tracking (delivery acks, roundtrip time) - this is not standardised by XMPP XEPs though, so implementations vary.
This being said, Postgres will probably only allow you to scale into hundreds of thousands of online users. It might require table sharding and user sharding across MongooseIM clusters using XMPP federation to allow for communication between all users. DB nodes in this architecture are associated with MIM clusters, therefore do not share data across cluster boundaries. Setups along these lines are known to work in production.
If you have to handle millions of users, you most likely have to switch from PostgreSQL to a database which can itself be more easily scaled. MongooseIM supports Cassandra, which currently seems to be the best bet. Personally, I'd curious to find out how TiKV (let me know, please, if you try that one) fares in such setups. It pretends to be MySQL on the wire, so should also work with MongooseIM, but operationally behaves similarly to Cassandra: automated replication, cluster scalability, fault tolerance, etc.
Ultimately, though, you'll only know by testing yourself. Instagram shares some interesting stories about Postgres at scale which show the amount of maintenance required. Alas, load tests reaching millions of users are prohibitively expensive due to cloud costs, so such test results are not common.
Thank erszcz, that was very informative. It seems like I must not under-estimate the potencial of Postgresql and should not blindly pick up any nosql tech. Although nosql like Cassandra surely can be a wise option. But if Instagram can manage their work in Postgresql then I must take it serious too
– Shubham1164
Jan 2 at 14:23
add a comment |
This is a complex matter. First of all, it depends a lot on the set of features you have enabled in MongooseIM, so any load tests done by others might not reflect your particular configuration and environment.
Moreover, to actually be able to monitor all the important parameters in a system reaching tens of thousands of users, you'll probably need client-side performance tracking (delivery acks, roundtrip time) - this is not standardised by XMPP XEPs though, so implementations vary.
This being said, Postgres will probably only allow you to scale into hundreds of thousands of online users. It might require table sharding and user sharding across MongooseIM clusters using XMPP federation to allow for communication between all users. DB nodes in this architecture are associated with MIM clusters, therefore do not share data across cluster boundaries. Setups along these lines are known to work in production.
If you have to handle millions of users, you most likely have to switch from PostgreSQL to a database which can itself be more easily scaled. MongooseIM supports Cassandra, which currently seems to be the best bet. Personally, I'd curious to find out how TiKV (let me know, please, if you try that one) fares in such setups. It pretends to be MySQL on the wire, so should also work with MongooseIM, but operationally behaves similarly to Cassandra: automated replication, cluster scalability, fault tolerance, etc.
Ultimately, though, you'll only know by testing yourself. Instagram shares some interesting stories about Postgres at scale which show the amount of maintenance required. Alas, load tests reaching millions of users are prohibitively expensive due to cloud costs, so such test results are not common.
This is a complex matter. First of all, it depends a lot on the set of features you have enabled in MongooseIM, so any load tests done by others might not reflect your particular configuration and environment.
Moreover, to actually be able to monitor all the important parameters in a system reaching tens of thousands of users, you'll probably need client-side performance tracking (delivery acks, roundtrip time) - this is not standardised by XMPP XEPs though, so implementations vary.
This being said, Postgres will probably only allow you to scale into hundreds of thousands of online users. It might require table sharding and user sharding across MongooseIM clusters using XMPP federation to allow for communication between all users. DB nodes in this architecture are associated with MIM clusters, therefore do not share data across cluster boundaries. Setups along these lines are known to work in production.
If you have to handle millions of users, you most likely have to switch from PostgreSQL to a database which can itself be more easily scaled. MongooseIM supports Cassandra, which currently seems to be the best bet. Personally, I'd curious to find out how TiKV (let me know, please, if you try that one) fares in such setups. It pretends to be MySQL on the wire, so should also work with MongooseIM, but operationally behaves similarly to Cassandra: automated replication, cluster scalability, fault tolerance, etc.
Ultimately, though, you'll only know by testing yourself. Instagram shares some interesting stories about Postgres at scale which show the amount of maintenance required. Alas, load tests reaching millions of users are prohibitively expensive due to cloud costs, so such test results are not common.
answered Jan 2 at 11:23
erszczerszcz
1,059715
1,059715
Thank erszcz, that was very informative. It seems like I must not under-estimate the potencial of Postgresql and should not blindly pick up any nosql tech. Although nosql like Cassandra surely can be a wise option. But if Instagram can manage their work in Postgresql then I must take it serious too
– Shubham1164
Jan 2 at 14:23
add a comment |
Thank erszcz, that was very informative. It seems like I must not under-estimate the potencial of Postgresql and should not blindly pick up any nosql tech. Although nosql like Cassandra surely can be a wise option. But if Instagram can manage their work in Postgresql then I must take it serious too
– Shubham1164
Jan 2 at 14:23
Thank erszcz, that was very informative. It seems like I must not under-estimate the potencial of Postgresql and should not blindly pick up any nosql tech. Although nosql like Cassandra surely can be a wise option. But if Instagram can manage their work in Postgresql then I must take it serious too
– Shubham1164
Jan 2 at 14:23
Thank erszcz, that was very informative. It seems like I must not under-estimate the potencial of Postgresql and should not blindly pick up any nosql tech. Although nosql like Cassandra surely can be a wise option. But if Instagram can manage their work in Postgresql then I must take it serious too
– Shubham1164
Jan 2 at 14:23
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53972068%2fhow-many-users-can-mongooseim-handle-on-the-postgresql-backend%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
ez,HWuYsJWhxerfWY,14I,Np4QyZ,57f,T,UFRakWfb,VaA 7NM1ZIvm Y7PBNq7WSdeHFAn4wxgXx,dd,D2ZyTGQ kA2Uiv3