Django Celery - Creating duplicate users due to mutiple workers
I am facing a weird issue with celery in production. Currently, the production server has 4 celery workers which handle all the tasks registered by my django app. No custom queues are defined. The workers are basically 4 separate supervisor conf files.
Now, in my app I am handling facebook webhook data, and I want a user with a specific FacebookID to be only created once on my backend. But, recently I checked and found out that there are users who have the same FacebookID, which should not have happened.
What happened I think was e.g. user with FacebookID 666 sent me a webhook data. Now, a task is created which will create a new user instance in my database with FacebookID 666. Now, before the user is created in my database, the user hits me with another data, which also created a task but under a different worker, and thus I got myself two users with the same FacebookID.
Is there any way I can configure celery to handle a user with a specific FacebookID to create tasks only in ONE worker? Or have I completely misjudged the situation over here?
django celery
asked Jan 1 at 10:39
MehranMehran
238110
|
show 2 more comments
I am facing a weird issue with celery in production. Currently, the production server has 4 celery workers which handle all the tasks registered by my django app. No custom queues are defined. The workers are basically 4 separate supervisor conf files.
Now, in my app I am handling facebook webhook data, and I want a user with a specific FacebookID to be only created once on my backend. But, recently I checked and found out that there are users who have the same FacebookID, which should not have happened.
What happened I think was e.g. user with FacebookID 666 sent me a webhook data. Now, a task is created which will create a new user instance in my database with FacebookID 666. Now, before the user is created in my database, the user hits me with another data, which also created a task but under a different worker, and thus I got myself two users with the same FacebookID.
Is there any way I can configure celery to handle a user with a specific FacebookID to create tasks only in ONE worker? Or have I completely misjudged the situation over here?
django celery
asked Jan 1 at 10:39
MehranMehran
238110
Disable the register button on click so that the user doesn't get to click it twice, as well as other general methods to avoid duplicate form submission (unique ids in hidden inputs etc.) - stackoverflow.com/questions/16814157/…
– Wiggy A.
Jan 1 at 10:55
What are you using for your broker?
– 2ps
Jan 2 at 2:38
@WiggyA. this is a facebook webhook, everytime a user messages on the page this data will hit my server
– Mehran
Jan 2 at 5:38
@2ps rabbitmq broker
– Mehran
Jan 2 at 5:38
what about setting ayour_facebook_id_keeper_fieldtounique=Truein your database?
– Chiefir
Jan 2 at 15:06
|
show 2 more comments
I am facing a weird issue with celery in production. Currently, the production server has 4 celery workers which handle all the tasks registered by my django app. No custom queues are defined. The workers are basically 4 separate supervisor conf files.
Now, in my app I am handling facebook webhook data, and I want a user with a specific FacebookID to be only created once on my backend. But, recently I checked and found out that there are users who have the same FacebookID, which should not have happened.
What happened I think was e.g. user with FacebookID 666 sent me a webhook data. Now, a task is created which will create a new user instance in my database with FacebookID 666. Now, before the user is created in my database, the user hits me with another data, which also created a task but under a different worker, and thus I got myself two users with the same FacebookID.
Is there any way I can configure celery to handle a user with a specific FacebookID to create tasks only in ONE worker? Or have I completely misjudged the situation over here?
django celery
asked Jan 1 at 10:39
MehranMehran
238110
I am facing a weird issue with celery in production. Currently, the production server has 4 celery workers which handle all the tasks registered by my django app. No custom queues are defined. The workers are basically 4 separate supervisor conf files.
Now, in my app I am handling facebook webhook data, and I want a user with a specific FacebookID to be only created once on my backend. But, recently I checked and found out that there are users who have the same FacebookID, which should not have happened.
What happened I think was e.g. user with FacebookID 666 sent me a webhook data. Now, a task is created which will create a new user instance in my database with FacebookID 666. Now, before the user is created in my database, the user hits me with another data, which also created a task but under a different worker, and thus I got myself two users with the same FacebookID.
Is there any way I can configure celery to handle a user with a specific FacebookID to create tasks only in ONE worker? Or have I completely misjudged the situation over here?
django celery
django celery
asked Jan 1 at 10:39
MehranMehran
238110
asked Jan 1 at 10:39
MehranMehran
238110
asked Jan 1 at 10:39
MehranMehran
238110
asked Jan 1 at 10:39
MehranMehran
238110
asked Jan 1 at 10:39
MehranMehran
238110
238110
Disable the register button on click so that the user doesn't get to click it twice, as well as other general methods to avoid duplicate form submission (unique ids in hidden inputs etc.) - stackoverflow.com/questions/16814157/…
– Wiggy A.
Jan 1 at 10:55
What are you using for your broker?
– 2ps
Jan 2 at 2:38
@WiggyA. this is a facebook webhook, everytime a user messages on the page this data will hit my server
– Mehran
Jan 2 at 5:38
@2ps rabbitmq broker
– Mehran
Jan 2 at 5:38
what about setting ayour_facebook_id_keeper_fieldtounique=Truein your database?
– Chiefir
Jan 2 at 15:06
|
show 2 more comments
Disable the register button on click so that the user doesn't get to click it twice, as well as other general methods to avoid duplicate form submission (unique ids in hidden inputs etc.) - stackoverflow.com/questions/16814157/…
– Wiggy A.
Jan 1 at 10:55
What are you using for your broker?
– 2ps
Jan 2 at 2:38
@WiggyA. this is a facebook webhook, everytime a user messages on the page this data will hit my server
– Mehran
Jan 2 at 5:38
@2ps rabbitmq broker
– Mehran
Jan 2 at 5:38
what about setting ayour_facebook_id_keeper_fieldtounique=Truein your database?
– Chiefir
Jan 2 at 15:06
Disable the register button on click so that the user doesn't get to click it twice, as well as other general methods to avoid duplicate form submission (unique ids in hidden inputs etc.) - stackoverflow.com/questions/16814157/…
– Wiggy A.
Jan 1 at 10:55
Disable the register button on click so that the user doesn't get to click it twice, as well as other general methods to avoid duplicate form submission (unique ids in hidden inputs etc.) - stackoverflow.com/questions/16814157/…
– Wiggy A.
Jan 1 at 10:55
What are you using for your broker?
– 2ps
Jan 2 at 2:38
What are you using for your broker?
– 2ps
Jan 2 at 2:38
@WiggyA. this is a facebook webhook, everytime a user messages on the page this data will hit my server
– Mehran
Jan 2 at 5:38
@WiggyA. this is a facebook webhook, everytime a user messages on the page this data will hit my server
– Mehran
Jan 2 at 5:38
@2ps rabbitmq broker
– Mehran
Jan 2 at 5:38
@2ps rabbitmq broker
– Mehran
Jan 2 at 5:38
what about setting a
your_facebook_id_keeper_field to unique=True in your database?– Chiefir
Jan 2 at 15:06
what about setting a
your_facebook_id_keeper_field to unique=True in your database?– Chiefir
Jan 2 at 15:06
|
show 2 more comments
1 Answer
1
active
oldest
votes
Essentially, you need a user-level distributed lock to prevent multiple workers from working on the same user. There are several ways to accomplish this, the most straightforward being a database such as mysql or redis. In mysql, the first process would transactionally (1) check for an existing row in a database table with the user ID (e.g., email or other unique identifier) and (2) if no row exists, creating that row; (3) and if a row exists, return early without doing anything. You can also do this in redis using a redlock or for smaller systems just using SETNX
answered Jan 3 at 6:40
2ps2ps
7,94221031
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53994796%2fdjango-celery-creating-duplicate-users-due-to-mutiple-workers%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Essentially, you need a user-level distributed lock to prevent multiple workers from working on the same user. There are several ways to accomplish this, the most straightforward being a database such as mysql or redis. In mysql, the first process would transactionally (1) check for an existing row in a database table with the user ID (e.g., email or other unique identifier) and (2) if no row exists, creating that row; (3) and if a row exists, return early without doing anything. You can also do this in redis using a redlock or for smaller systems just using SETNX
answered Jan 3 at 6:40
2ps2ps
7,94221031
add a comment |
Essentially, you need a user-level distributed lock to prevent multiple workers from working on the same user. There are several ways to accomplish this, the most straightforward being a database such as mysql or redis. In mysql, the first process would transactionally (1) check for an existing row in a database table with the user ID (e.g., email or other unique identifier) and (2) if no row exists, creating that row; (3) and if a row exists, return early without doing anything. You can also do this in redis using a redlock or for smaller systems just using SETNX
answered Jan 3 at 6:40
2ps2ps
7,94221031
add a comment |
Essentially, you need a user-level distributed lock to prevent multiple workers from working on the same user. There are several ways to accomplish this, the most straightforward being a database such as mysql or redis. In mysql, the first process would transactionally (1) check for an existing row in a database table with the user ID (e.g., email or other unique identifier) and (2) if no row exists, creating that row; (3) and if a row exists, return early without doing anything. You can also do this in redis using a redlock or for smaller systems just using SETNX
answered Jan 3 at 6:40
2ps2ps
7,94221031
Essentially, you need a user-level distributed lock to prevent multiple workers from working on the same user. There are several ways to accomplish this, the most straightforward being a database such as mysql or redis. In mysql, the first process would transactionally (1) check for an existing row in a database table with the user ID (e.g., email or other unique identifier) and (2) if no row exists, creating that row; (3) and if a row exists, return early without doing anything. You can also do this in redis using a redlock or for smaller systems just using SETNX
answered Jan 3 at 6:40
2ps2ps
7,94221031
answered Jan 3 at 6:40
2ps2ps
7,94221031
answered Jan 3 at 6:40
2ps2ps
7,94221031
answered Jan 3 at 6:40
2ps2ps
7,94221031
7,94221031
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53994796%2fdjango-celery-creating-duplicate-users-due-to-mutiple-workers%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Disable the register button on click so that the user doesn't get to click it twice, as well as other general methods to avoid duplicate form submission (unique ids in hidden inputs etc.) - stackoverflow.com/questions/16814157/…
– Wiggy A.
Jan 1 at 10:55
What are you using for your broker?
– 2ps
Jan 2 at 2:38
@WiggyA. this is a facebook webhook, everytime a user messages on the page this data will hit my server
– Mehran
Jan 2 at 5:38
@2ps rabbitmq broker
– Mehran
Jan 2 at 5:38
what about setting a
your_facebook_id_keeper_fieldtounique=Truein your database?– Chiefir
Jan 2 at 15:06