Designing distributed application in C
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
Currently I have a monolithic process written in C performing following set of operations
a. Accept data from user (in certain payload format).
IF the user has provided data first time validate if it is standard
compliant and store it in user data-store
ELSE compare the data provided by user previously which is stored in user
data-store and check if it is duplicate or not.
IF DUPLICATE Skip it
ELSE Validate the data provided by the user and update previous information
in user data-store.
b. Whenever user data-store is updated,
initialize analytics app to process the information stored in user
date-store and generate final output
Replicate the updates into backup user data-store for future reference.
Currently I am facing scale issues especially with the rate at which user keeps inputting data. As a result current application has the limitation of accepting only certain amount of user data requests with one monolithic process.
Adding pthreads would be the next step to scale the application which sometimes adds complexity to the codebase and proper care has to be taken with respect to locking for consistency.
I want to try out microservices approach where I want to divide the current monolithic process into multiple microservices as mentioned below so that I can improve each process separately improving its performance for future updates.
Process-01: Contains logic to accept data from user and does all standard verification's and update the shared data-store.
Process-02: whenever the shared data-store is updated, contains logic to process the data in it and create final output.
Process-03: Takes care of replicating share-datastore.
process-01 ==== (shared-datastore) === process-02
||
process-03
It seems I need a shared data-store which process-01 and process-02 would request the data provided by user for processing. Would this create a bottleneck as I need to create a copy of the requested as a message and share it with processes ? For process-02 I need to create the whole datastore as copy so that it can work on the data independently.
Ideally process-01 needs both read and write permissions while process-02 needs only read permission of the data in shared data-store.
Since we are using C for our application, please suggest if the planned new design has any more caveats? Is there any better design for the same ?
Even any other design suggestions would be of great help. Pointing to similar application design in C would also be helpful.
c shared-memory distributed-system
asked Jan 4 at 0:14
codingfreakcodingfreak
1,96193855
add a comment |
Currently I have a monolithic process written in C performing following set of operations
a. Accept data from user (in certain payload format).
IF the user has provided data first time validate if it is standard
compliant and store it in user data-store
ELSE compare the data provided by user previously which is stored in user
data-store and check if it is duplicate or not.
IF DUPLICATE Skip it
ELSE Validate the data provided by the user and update previous information
in user data-store.
b. Whenever user data-store is updated,
initialize analytics app to process the information stored in user
date-store and generate final output
Replicate the updates into backup user data-store for future reference.
Currently I am facing scale issues especially with the rate at which user keeps inputting data. As a result current application has the limitation of accepting only certain amount of user data requests with one monolithic process.
Adding pthreads would be the next step to scale the application which sometimes adds complexity to the codebase and proper care has to be taken with respect to locking for consistency.
I want to try out microservices approach where I want to divide the current monolithic process into multiple microservices as mentioned below so that I can improve each process separately improving its performance for future updates.
Process-01: Contains logic to accept data from user and does all standard verification's and update the shared data-store.
Process-02: whenever the shared data-store is updated, contains logic to process the data in it and create final output.
Process-03: Takes care of replicating share-datastore.
process-01 ==== (shared-datastore) === process-02
||
process-03
It seems I need a shared data-store which process-01 and process-02 would request the data provided by user for processing. Would this create a bottleneck as I need to create a copy of the requested as a message and share it with processes ? For process-02 I need to create the whole datastore as copy so that it can work on the data independently.
Ideally process-01 needs both read and write permissions while process-02 needs only read permission of the data in shared data-store.
Since we are using C for our application, please suggest if the planned new design has any more caveats? Is there any better design for the same ?
Even any other design suggestions would be of great help. Pointing to similar application design in C would also be helpful.
c shared-memory distributed-system
asked Jan 4 at 0:14
codingfreakcodingfreak
1,96193855
Your questions are too broad and with very little information. So, there is no good answer in this forum. As a suggestion, start with defining which part of your app cause the bottleneck. It looks like your scheme is sequential by nature and will not give you any gain. Think about parallelizing some parts. Also, it sounds like you will be limited by the disk IO in any case. Think how to avoid extra disk operations.
– Serge
Jan 4 at 2:37
You are asking (multiple) architecture questions. The stackoverflow Q&A format is not a great venue for answering your questions. Perhaps as @Serge says, you could decompose your problem into a series of questions?
– ChuckCottrill
Jan 4 at 5:10
add a comment |
Currently I have a monolithic process written in C performing following set of operations
a. Accept data from user (in certain payload format).
IF the user has provided data first time validate if it is standard
compliant and store it in user data-store
ELSE compare the data provided by user previously which is stored in user
data-store and check if it is duplicate or not.
IF DUPLICATE Skip it
ELSE Validate the data provided by the user and update previous information
in user data-store.
b. Whenever user data-store is updated,
initialize analytics app to process the information stored in user
date-store and generate final output
Replicate the updates into backup user data-store for future reference.
Currently I am facing scale issues especially with the rate at which user keeps inputting data. As a result current application has the limitation of accepting only certain amount of user data requests with one monolithic process.
Adding pthreads would be the next step to scale the application which sometimes adds complexity to the codebase and proper care has to be taken with respect to locking for consistency.
I want to try out microservices approach where I want to divide the current monolithic process into multiple microservices as mentioned below so that I can improve each process separately improving its performance for future updates.
Process-01: Contains logic to accept data from user and does all standard verification's and update the shared data-store.
Process-02: whenever the shared data-store is updated, contains logic to process the data in it and create final output.
Process-03: Takes care of replicating share-datastore.
process-01 ==== (shared-datastore) === process-02
||
process-03
It seems I need a shared data-store which process-01 and process-02 would request the data provided by user for processing. Would this create a bottleneck as I need to create a copy of the requested as a message and share it with processes ? For process-02 I need to create the whole datastore as copy so that it can work on the data independently.
Ideally process-01 needs both read and write permissions while process-02 needs only read permission of the data in shared data-store.
Since we are using C for our application, please suggest if the planned new design has any more caveats? Is there any better design for the same ?
Even any other design suggestions would be of great help. Pointing to similar application design in C would also be helpful.
c shared-memory distributed-system
asked Jan 4 at 0:14
codingfreakcodingfreak
1,96193855
Currently I have a monolithic process written in C performing following set of operations
a. Accept data from user (in certain payload format).
IF the user has provided data first time validate if it is standard
compliant and store it in user data-store
ELSE compare the data provided by user previously which is stored in user
data-store and check if it is duplicate or not.
IF DUPLICATE Skip it
ELSE Validate the data provided by the user and update previous information
in user data-store.
b. Whenever user data-store is updated,
initialize analytics app to process the information stored in user
date-store and generate final output
Replicate the updates into backup user data-store for future reference.
Currently I am facing scale issues especially with the rate at which user keeps inputting data. As a result current application has the limitation of accepting only certain amount of user data requests with one monolithic process.
Adding pthreads would be the next step to scale the application which sometimes adds complexity to the codebase and proper care has to be taken with respect to locking for consistency.
I want to try out microservices approach where I want to divide the current monolithic process into multiple microservices as mentioned below so that I can improve each process separately improving its performance for future updates.
Process-01: Contains logic to accept data from user and does all standard verification's and update the shared data-store.
Process-02: whenever the shared data-store is updated, contains logic to process the data in it and create final output.
Process-03: Takes care of replicating share-datastore.
process-01 ==== (shared-datastore) === process-02
||
process-03
It seems I need a shared data-store which process-01 and process-02 would request the data provided by user for processing. Would this create a bottleneck as I need to create a copy of the requested as a message and share it with processes ? For process-02 I need to create the whole datastore as copy so that it can work on the data independently.
Ideally process-01 needs both read and write permissions while process-02 needs only read permission of the data in shared data-store.
Since we are using C for our application, please suggest if the planned new design has any more caveats? Is there any better design for the same ?
Even any other design suggestions would be of great help. Pointing to similar application design in C would also be helpful.
c shared-memory distributed-system
c shared-memory distributed-system
asked Jan 4 at 0:14
codingfreakcodingfreak
1,96193855
asked Jan 4 at 0:14
codingfreakcodingfreak
1,96193855
asked Jan 4 at 0:14
codingfreakcodingfreak
1,96193855
asked Jan 4 at 0:14
codingfreakcodingfreak
1,96193855
asked Jan 4 at 0:14
codingfreakcodingfreak
1,96193855
1,96193855
Your questions are too broad and with very little information. So, there is no good answer in this forum. As a suggestion, start with defining which part of your app cause the bottleneck. It looks like your scheme is sequential by nature and will not give you any gain. Think about parallelizing some parts. Also, it sounds like you will be limited by the disk IO in any case. Think how to avoid extra disk operations.
– Serge
Jan 4 at 2:37
You are asking (multiple) architecture questions. The stackoverflow Q&A format is not a great venue for answering your questions. Perhaps as @Serge says, you could decompose your problem into a series of questions?
– ChuckCottrill
Jan 4 at 5:10
add a comment |
Your questions are too broad and with very little information. So, there is no good answer in this forum. As a suggestion, start with defining which part of your app cause the bottleneck. It looks like your scheme is sequential by nature and will not give you any gain. Think about parallelizing some parts. Also, it sounds like you will be limited by the disk IO in any case. Think how to avoid extra disk operations.
– Serge
Jan 4 at 2:37
You are asking (multiple) architecture questions. The stackoverflow Q&A format is not a great venue for answering your questions. Perhaps as @Serge says, you could decompose your problem into a series of questions?
– ChuckCottrill
Jan 4 at 5:10
Your questions are too broad and with very little information. So, there is no good answer in this forum. As a suggestion, start with defining which part of your app cause the bottleneck. It looks like your scheme is sequential by nature and will not give you any gain. Think about parallelizing some parts. Also, it sounds like you will be limited by the disk IO in any case. Think how to avoid extra disk operations.
– Serge
Jan 4 at 2:37
Your questions are too broad and with very little information. So, there is no good answer in this forum. As a suggestion, start with defining which part of your app cause the bottleneck. It looks like your scheme is sequential by nature and will not give you any gain. Think about parallelizing some parts. Also, it sounds like you will be limited by the disk IO in any case. Think how to avoid extra disk operations.
– Serge
Jan 4 at 2:37
You are asking (multiple) architecture questions. The stackoverflow Q&A format is not a great venue for answering your questions. Perhaps as @Serge says, you could decompose your problem into a series of questions?
– ChuckCottrill
Jan 4 at 5:10
You are asking (multiple) architecture questions. The stackoverflow Q&A format is not a great venue for answering your questions. Perhaps as @Serge says, you could decompose your problem into a series of questions?
– ChuckCottrill
Jan 4 at 5:10
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54031574%2fdesigning-distributed-application-in-c%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54031574%2fdesigning-distributed-application-in-c%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Your questions are too broad and with very little information. So, there is no good answer in this forum. As a suggestion, start with defining which part of your app cause the bottleneck. It looks like your scheme is sequential by nature and will not give you any gain. Think about parallelizing some parts. Also, it sounds like you will be limited by the disk IO in any case. Think how to avoid extra disk operations.
– Serge
Jan 4 at 2:37
You are asking (multiple) architecture questions. The stackoverflow Q&A format is not a great venue for answering your questions. Perhaps as @Serge says, you could decompose your problem into a series of questions?
– ChuckCottrill
Jan 4 at 5:10