Scheduled Task Produces Too Many Tasks
In one of my ECS clusters I have a scheduled Fargate task that's meant to spin up 8 instances of it's given target. However, when the task procs it starts up waaayyyy more than 8 tasks. Sometimes as many as 50. Does anyone know what could be causing this to happen?
Details:
- Cron Expression:
cron(40 16 ? * 1-5 *)
- Target Definition:
amazon-web-services amazon-ecs
asked Jan 2 at 17:07
ValevalorinValevalorin
311311
add a comment |
In one of my ECS clusters I have a scheduled Fargate task that's meant to spin up 8 instances of it's given target. However, when the task procs it starts up waaayyyy more than 8 tasks. Sometimes as many as 50. Does anyone know what could be causing this to happen?
Details:
- Cron Expression:
cron(40 16 ? * 1-5 *)
- Target Definition:
amazon-web-services amazon-ecs
asked Jan 2 at 17:07
ValevalorinValevalorin
311311
well i was just validating your cron expression i have found that '?' can only be specfied for Day-of-Month or Day-of-Week and your cron expression is invalid because you have applied it for hour, can it be an issue ?
– varnit
Jan 2 at 18:39
It's not a 'real' cron expression that's used for scheduled tasks. It's some bastardization that Amazon came up with. Format here:cron(Minutes Hours Day-of-month Month Day-of-week Year). Thanks for the attempt though.
– Valevalorin
Jan 2 at 20:35
add a comment |
In one of my ECS clusters I have a scheduled Fargate task that's meant to spin up 8 instances of it's given target. However, when the task procs it starts up waaayyyy more than 8 tasks. Sometimes as many as 50. Does anyone know what could be causing this to happen?
Details:
- Cron Expression:
cron(40 16 ? * 1-5 *)
- Target Definition:
amazon-web-services amazon-ecs
asked Jan 2 at 17:07
ValevalorinValevalorin
311311
In one of my ECS clusters I have a scheduled Fargate task that's meant to spin up 8 instances of it's given target. However, when the task procs it starts up waaayyyy more than 8 tasks. Sometimes as many as 50. Does anyone know what could be causing this to happen?
Details:
- Cron Expression:
cron(40 16 ? * 1-5 *)
- Target Definition:
amazon-web-services amazon-ecs
amazon-web-services amazon-ecs
asked Jan 2 at 17:07
ValevalorinValevalorin
311311
asked Jan 2 at 17:07
ValevalorinValevalorin
311311
asked Jan 2 at 17:07
ValevalorinValevalorin
311311
asked Jan 2 at 17:07
ValevalorinValevalorin
311311
asked Jan 2 at 17:07
ValevalorinValevalorin
311311
311311
well i was just validating your cron expression i have found that '?' can only be specfied for Day-of-Month or Day-of-Week and your cron expression is invalid because you have applied it for hour, can it be an issue ?
– varnit
Jan 2 at 18:39
It's not a 'real' cron expression that's used for scheduled tasks. It's some bastardization that Amazon came up with. Format here:cron(Minutes Hours Day-of-month Month Day-of-week Year). Thanks for the attempt though.
– Valevalorin
Jan 2 at 20:35
add a comment |
well i was just validating your cron expression i have found that '?' can only be specfied for Day-of-Month or Day-of-Week and your cron expression is invalid because you have applied it for hour, can it be an issue ?
– varnit
Jan 2 at 18:39
It's not a 'real' cron expression that's used for scheduled tasks. It's some bastardization that Amazon came up with. Format here:cron(Minutes Hours Day-of-month Month Day-of-week Year). Thanks for the attempt though.
– Valevalorin
Jan 2 at 20:35
well i was just validating your cron expression i have found that '?' can only be specfied for Day-of-Month or Day-of-Week and your cron expression is invalid because you have applied it for hour, can it be an issue ?
– varnit
Jan 2 at 18:39
well i was just validating your cron expression i have found that '?' can only be specfied for Day-of-Month or Day-of-Week and your cron expression is invalid because you have applied it for hour, can it be an issue ?
– varnit
Jan 2 at 18:39
It's not a 'real' cron expression that's used for scheduled tasks. It's some bastardization that Amazon came up with. Format here:
cron(Minutes Hours Day-of-month Month Day-of-week Year). Thanks for the attempt though.– Valevalorin
Jan 2 at 20:35
It's not a 'real' cron expression that's used for scheduled tasks. It's some bastardization that Amazon came up with. Format here:
cron(Minutes Hours Day-of-month Month Day-of-week Year). Thanks for the attempt though.– Valevalorin
Jan 2 at 20:35
add a comment |
1 Answer
1
active
oldest
votes
For anyone who might run into this problem in the future:
This problem occurred because we had too many tasks running the cluster. As of the writing of this answer AWS set of limit of 50 tasks running in a single cluster. Before the rule triggered there was already close to 50 tasks running. The rule would proc and would start spinning up new tasks trying to get to the desired number (8).
However, due to the limit it would never be able to get 8 because new tasks over the limit would just get shutdown. So it would keep trying, and keep trying, and keep trying to spin up tasks which led to there being a huge pending queue of tasks that would seemingly push (nearly) all of our tasks out of the cluster and we'd be left with way more tasks than we had asked for.
The solution: we just moved the scheduled task into a new cluster to avoid the 50 task limit.
answered Jan 3 at 18:59
ValevalorinValevalorin
311311
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54010383%2fscheduled-task-produces-too-many-tasks%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
For anyone who might run into this problem in the future:
This problem occurred because we had too many tasks running the cluster. As of the writing of this answer AWS set of limit of 50 tasks running in a single cluster. Before the rule triggered there was already close to 50 tasks running. The rule would proc and would start spinning up new tasks trying to get to the desired number (8).
However, due to the limit it would never be able to get 8 because new tasks over the limit would just get shutdown. So it would keep trying, and keep trying, and keep trying to spin up tasks which led to there being a huge pending queue of tasks that would seemingly push (nearly) all of our tasks out of the cluster and we'd be left with way more tasks than we had asked for.
The solution: we just moved the scheduled task into a new cluster to avoid the 50 task limit.
answered Jan 3 at 18:59
ValevalorinValevalorin
311311
add a comment |
For anyone who might run into this problem in the future:
This problem occurred because we had too many tasks running the cluster. As of the writing of this answer AWS set of limit of 50 tasks running in a single cluster. Before the rule triggered there was already close to 50 tasks running. The rule would proc and would start spinning up new tasks trying to get to the desired number (8).
However, due to the limit it would never be able to get 8 because new tasks over the limit would just get shutdown. So it would keep trying, and keep trying, and keep trying to spin up tasks which led to there being a huge pending queue of tasks that would seemingly push (nearly) all of our tasks out of the cluster and we'd be left with way more tasks than we had asked for.
The solution: we just moved the scheduled task into a new cluster to avoid the 50 task limit.
answered Jan 3 at 18:59
ValevalorinValevalorin
311311
add a comment |
For anyone who might run into this problem in the future:
This problem occurred because we had too many tasks running the cluster. As of the writing of this answer AWS set of limit of 50 tasks running in a single cluster. Before the rule triggered there was already close to 50 tasks running. The rule would proc and would start spinning up new tasks trying to get to the desired number (8).
However, due to the limit it would never be able to get 8 because new tasks over the limit would just get shutdown. So it would keep trying, and keep trying, and keep trying to spin up tasks which led to there being a huge pending queue of tasks that would seemingly push (nearly) all of our tasks out of the cluster and we'd be left with way more tasks than we had asked for.
The solution: we just moved the scheduled task into a new cluster to avoid the 50 task limit.
answered Jan 3 at 18:59
ValevalorinValevalorin
311311
For anyone who might run into this problem in the future:
This problem occurred because we had too many tasks running the cluster. As of the writing of this answer AWS set of limit of 50 tasks running in a single cluster. Before the rule triggered there was already close to 50 tasks running. The rule would proc and would start spinning up new tasks trying to get to the desired number (8).
However, due to the limit it would never be able to get 8 because new tasks over the limit would just get shutdown. So it would keep trying, and keep trying, and keep trying to spin up tasks which led to there being a huge pending queue of tasks that would seemingly push (nearly) all of our tasks out of the cluster and we'd be left with way more tasks than we had asked for.
The solution: we just moved the scheduled task into a new cluster to avoid the 50 task limit.
answered Jan 3 at 18:59
ValevalorinValevalorin
311311
answered Jan 3 at 18:59
ValevalorinValevalorin
311311
answered Jan 3 at 18:59
ValevalorinValevalorin
311311
answered Jan 3 at 18:59
ValevalorinValevalorin
311311
311311
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54010383%2fscheduled-task-produces-too-many-tasks%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
well i was just validating your cron expression i have found that '?' can only be specfied for Day-of-Month or Day-of-Week and your cron expression is invalid because you have applied it for hour, can it be an issue ?
– varnit
Jan 2 at 18:39
It's not a 'real' cron expression that's used for scheduled tasks. It's some bastardization that Amazon came up with. Format here:
cron(Minutes Hours Day-of-month Month Day-of-week Year). Thanks for the attempt though.– Valevalorin
Jan 2 at 20:35