Laravel 5.7 - Queues Jobs are too slow
I use Laravel 5.7 and 3 queues jobs, the time between jobs is too long/slow.
I foreach items of RSS feeds in the first job, and I dispatch this item in second job, etc... I don't enter in details but there are some ridiculous little calculations that must not take time.
The problem is that every dispatch to a job takes a lot of time. Horizon and Telescope do not allow me to debug.
The machine I use has 32 GB of RAM, and there are several processes (15 each) that turn the tails.
[program:mywebsite_feeder]
command=/RunCloud/Packages/php72rc/bin/php artisan queue:work redis --queue=feeder --tries=3 --sleep=0
directory=/home/runcloud/webapps/mywebsite
redirect_stderr=true
autostart=true
autorestart=true
user=runcloud
numprocs=15
process_name=%(program_name)s_%(process_num)s
I have this error in laravel.log:
production.ERROR: AppJobsFeederJob has been attempted too many times
or run too long. The job may have previously timed out.
php laravel redis queue supervisord
asked Jan 2 at 14:50
pirmaxpirmax
457420
add a comment |
I use Laravel 5.7 and 3 queues jobs, the time between jobs is too long/slow.
I foreach items of RSS feeds in the first job, and I dispatch this item in second job, etc... I don't enter in details but there are some ridiculous little calculations that must not take time.
The problem is that every dispatch to a job takes a lot of time. Horizon and Telescope do not allow me to debug.
The machine I use has 32 GB of RAM, and there are several processes (15 each) that turn the tails.
[program:mywebsite_feeder]
command=/RunCloud/Packages/php72rc/bin/php artisan queue:work redis --queue=feeder --tries=3 --sleep=0
directory=/home/runcloud/webapps/mywebsite
redirect_stderr=true
autostart=true
autorestart=true
user=runcloud
numprocs=15
process_name=%(program_name)s_%(process_num)s
I have this error in laravel.log:
production.ERROR: AppJobsFeederJob has been attempted too many times
or run too long. The job may have previously timed out.
php laravel redis queue supervisord
asked Jan 2 at 14:50
pirmaxpirmax
457420
Are you using "sync" queue driver? For debugging failing jobs it's good to try / catch with logging exception details to the log.
– Chris Cynarski
Jan 2 at 17:15
I don't want use sync, but use Redis.
– pirmax
Jan 2 at 17:17
If dispatch takes long time and the job supposed to take long time it might suggest that the job is run synchronously not dispatched to the queue. Check dd(env('QUEUE_DRIVER')) somewhere in your code.
– Chris Cynarski
Jan 2 at 17:20
It's give me : redis, not sync.
– pirmax
Jan 2 at 17:30
Are you calling ::dispatch or ::dispatchNow ? It would be hard to find out what is wrong without the snippet of the code. I use redis for my queues and dispatching to the queue is instant.
– Chris Cynarski
Jan 2 at 17:46
add a comment |
I use Laravel 5.7 and 3 queues jobs, the time between jobs is too long/slow.
I foreach items of RSS feeds in the first job, and I dispatch this item in second job, etc... I don't enter in details but there are some ridiculous little calculations that must not take time.
The problem is that every dispatch to a job takes a lot of time. Horizon and Telescope do not allow me to debug.
The machine I use has 32 GB of RAM, and there are several processes (15 each) that turn the tails.
[program:mywebsite_feeder]
command=/RunCloud/Packages/php72rc/bin/php artisan queue:work redis --queue=feeder --tries=3 --sleep=0
directory=/home/runcloud/webapps/mywebsite
redirect_stderr=true
autostart=true
autorestart=true
user=runcloud
numprocs=15
process_name=%(program_name)s_%(process_num)s
I have this error in laravel.log:
production.ERROR: AppJobsFeederJob has been attempted too many times
or run too long. The job may have previously timed out.
php laravel redis queue supervisord
asked Jan 2 at 14:50
pirmaxpirmax
457420
I use Laravel 5.7 and 3 queues jobs, the time between jobs is too long/slow.
I foreach items of RSS feeds in the first job, and I dispatch this item in second job, etc... I don't enter in details but there are some ridiculous little calculations that must not take time.
The problem is that every dispatch to a job takes a lot of time. Horizon and Telescope do not allow me to debug.
The machine I use has 32 GB of RAM, and there are several processes (15 each) that turn the tails.
[program:mywebsite_feeder]
command=/RunCloud/Packages/php72rc/bin/php artisan queue:work redis --queue=feeder --tries=3 --sleep=0
directory=/home/runcloud/webapps/mywebsite
redirect_stderr=true
autostart=true
autorestart=true
user=runcloud
numprocs=15
process_name=%(program_name)s_%(process_num)s
I have this error in laravel.log:
production.ERROR: AppJobsFeederJob has been attempted too many times
or run too long. The job may have previously timed out.
php laravel redis queue supervisord
php laravel redis queue supervisord
asked Jan 2 at 14:50
pirmaxpirmax
457420
asked Jan 2 at 14:50
pirmaxpirmax
457420
edited Jan 2 at 15:00
pirmax
asked Jan 2 at 14:50
pirmaxpirmax
457420
asked Jan 2 at 14:50
pirmaxpirmax
457420
asked Jan 2 at 14:50
pirmaxpirmax
457420
457420
Are you using "sync" queue driver? For debugging failing jobs it's good to try / catch with logging exception details to the log.
– Chris Cynarski
Jan 2 at 17:15
I don't want use sync, but use Redis.
– pirmax
Jan 2 at 17:17
If dispatch takes long time and the job supposed to take long time it might suggest that the job is run synchronously not dispatched to the queue. Check dd(env('QUEUE_DRIVER')) somewhere in your code.
– Chris Cynarski
Jan 2 at 17:20
It's give me : redis, not sync.
– pirmax
Jan 2 at 17:30
Are you calling ::dispatch or ::dispatchNow ? It would be hard to find out what is wrong without the snippet of the code. I use redis for my queues and dispatching to the queue is instant.
– Chris Cynarski
Jan 2 at 17:46
add a comment |
Are you using "sync" queue driver? For debugging failing jobs it's good to try / catch with logging exception details to the log.
– Chris Cynarski
Jan 2 at 17:15
I don't want use sync, but use Redis.
– pirmax
Jan 2 at 17:17
If dispatch takes long time and the job supposed to take long time it might suggest that the job is run synchronously not dispatched to the queue. Check dd(env('QUEUE_DRIVER')) somewhere in your code.
– Chris Cynarski
Jan 2 at 17:20
It's give me : redis, not sync.
– pirmax
Jan 2 at 17:30
Are you calling ::dispatch or ::dispatchNow ? It would be hard to find out what is wrong without the snippet of the code. I use redis for my queues and dispatching to the queue is instant.
– Chris Cynarski
Jan 2 at 17:46
Are you using "sync" queue driver? For debugging failing jobs it's good to try / catch with logging exception details to the log.
– Chris Cynarski
Jan 2 at 17:15
Are you using "sync" queue driver? For debugging failing jobs it's good to try / catch with logging exception details to the log.
– Chris Cynarski
Jan 2 at 17:15
I don't want use sync, but use Redis.
– pirmax
Jan 2 at 17:17
I don't want use sync, but use Redis.
– pirmax
Jan 2 at 17:17
If dispatch takes long time and the job supposed to take long time it might suggest that the job is run synchronously not dispatched to the queue. Check dd(env('QUEUE_DRIVER')) somewhere in your code.
– Chris Cynarski
Jan 2 at 17:20
If dispatch takes long time and the job supposed to take long time it might suggest that the job is run synchronously not dispatched to the queue. Check dd(env('QUEUE_DRIVER')) somewhere in your code.
– Chris Cynarski
Jan 2 at 17:20
It's give me : redis, not sync.
– pirmax
Jan 2 at 17:30
It's give me : redis, not sync.
– pirmax
Jan 2 at 17:30
Are you calling ::dispatch or ::dispatchNow ? It would be hard to find out what is wrong without the snippet of the code. I use redis for my queues and dispatching to the queue is instant.
– Chris Cynarski
Jan 2 at 17:46
Are you calling ::dispatch or ::dispatchNow ? It would be hard to find out what is wrong without the snippet of the code. I use redis for my queues and dispatching to the queue is instant.
– Chris Cynarski
Jan 2 at 17:46
add a comment |
1 Answer
1
active
oldest
votes
I had the same issue and did a lot of searches but nothing help, even there is some issues about this bug in horizon Github but without a useful solution. the problem is about horizon and Redis bug for heavy tasks.
finally, I switch from Redis and horizon to SQL database (whatever you use in your project for me mssql) as queue connection and it fixed the problem
notice: use --timeout=0 in your artisan command
answered Jan 2 at 18:16
Milad TeimouriMilad Teimouri
35316
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54008408%2flaravel-5-7-queues-jobs-are-too-slow%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I had the same issue and did a lot of searches but nothing help, even there is some issues about this bug in horizon Github but without a useful solution. the problem is about horizon and Redis bug for heavy tasks.
finally, I switch from Redis and horizon to SQL database (whatever you use in your project for me mssql) as queue connection and it fixed the problem
notice: use --timeout=0 in your artisan command
answered Jan 2 at 18:16
Milad TeimouriMilad Teimouri
35316
add a comment |
I had the same issue and did a lot of searches but nothing help, even there is some issues about this bug in horizon Github but without a useful solution. the problem is about horizon and Redis bug for heavy tasks.
finally, I switch from Redis and horizon to SQL database (whatever you use in your project for me mssql) as queue connection and it fixed the problem
notice: use --timeout=0 in your artisan command
answered Jan 2 at 18:16
Milad TeimouriMilad Teimouri
35316
add a comment |
I had the same issue and did a lot of searches but nothing help, even there is some issues about this bug in horizon Github but without a useful solution. the problem is about horizon and Redis bug for heavy tasks.
finally, I switch from Redis and horizon to SQL database (whatever you use in your project for me mssql) as queue connection and it fixed the problem
notice: use --timeout=0 in your artisan command
answered Jan 2 at 18:16
Milad TeimouriMilad Teimouri
35316
I had the same issue and did a lot of searches but nothing help, even there is some issues about this bug in horizon Github but without a useful solution. the problem is about horizon and Redis bug for heavy tasks.
finally, I switch from Redis and horizon to SQL database (whatever you use in your project for me mssql) as queue connection and it fixed the problem
notice: use --timeout=0 in your artisan command
answered Jan 2 at 18:16
Milad TeimouriMilad Teimouri
35316
edited Jan 2 at 18:21
answered Jan 2 at 18:16
Milad TeimouriMilad Teimouri
35316
answered Jan 2 at 18:16
Milad TeimouriMilad Teimouri
35316
answered Jan 2 at 18:16
Milad TeimouriMilad Teimouri
35316
35316
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54008408%2flaravel-5-7-queues-jobs-are-too-slow%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Are you using "sync" queue driver? For debugging failing jobs it's good to try / catch with logging exception details to the log.
– Chris Cynarski
Jan 2 at 17:15
I don't want use sync, but use Redis.
– pirmax
Jan 2 at 17:17
If dispatch takes long time and the job supposed to take long time it might suggest that the job is run synchronously not dispatched to the queue. Check dd(env('QUEUE_DRIVER')) somewhere in your code.
– Chris Cynarski
Jan 2 at 17:20
It's give me : redis, not sync.
– pirmax
Jan 2 at 17:30
Are you calling ::dispatch or ::dispatchNow ? It would be hard to find out what is wrong without the snippet of the code. I use redis for my queues and dispatching to the queue is instant.
– Chris Cynarski
Jan 2 at 17:46