Spark - why application slower with more executors?
1
I have 2 executors and they are using 4 cores each with 3gb memory i've got an answer in 1.9 min but when I'm giving num of executors 8 with one core each it gives me answer in 12 mins, why the result is so different?
apache-spark apache-spark-sql
edited Jan 3 at 15:00
Robert Andrzejuk
3,04921528
asked Jan 3 at 13:50
Fahad KhanFahad Khan
63
add a comment |
1
I have 2 executors and they are using 4 cores each with 3gb memory i've got an answer in 1.9 min but when I'm giving num of executors 8 with one core each it gives me answer in 12 mins, why the result is so different?
apache-spark apache-spark-sql
edited Jan 3 at 15:00
Robert Andrzejuk
3,04921528
asked Jan 3 at 13:50
Fahad KhanFahad Khan
63
Can you say a bit more of what you are doing specifically? It could be that your data is good enough with 2 executors and adding executors is just spreading data more.
– Alessandro
Jan 3 at 15:46
I was using mysql database and in that I have 5.7 GB data and I was reading the data via spark. When I was running a query on it via spark it gave me the result in 1.9 minutes(2 executors, 4 cores/executor, executor memory 3gb, partitions 8) but when I increase the number of executors(8 executors, 1 core/executor, 1gb of executor memory, 8 partitions) so I got the query result in 12 minutes. I was wondering why it is so different as it is the same thing I guess.
– Fahad Khan
Jan 4 at 14:38
add a comment |
1
1
1
I have 2 executors and they are using 4 cores each with 3gb memory i've got an answer in 1.9 min but when I'm giving num of executors 8 with one core each it gives me answer in 12 mins, why the result is so different?
apache-spark apache-spark-sql
edited Jan 3 at 15:00
Robert Andrzejuk
3,04921528
asked Jan 3 at 13:50
Fahad KhanFahad Khan
63
I have 2 executors and they are using 4 cores each with 3gb memory i've got an answer in 1.9 min but when I'm giving num of executors 8 with one core each it gives me answer in 12 mins, why the result is so different?
apache-spark apache-spark-sql
apache-spark apache-spark-sql
edited Jan 3 at 15:00
Robert Andrzejuk
3,04921528
asked Jan 3 at 13:50
Fahad KhanFahad Khan
63
edited Jan 3 at 15:00
Robert Andrzejuk
3,04921528
asked Jan 3 at 13:50
Fahad KhanFahad Khan
63
edited Jan 3 at 15:00
Robert Andrzejuk
3,04921528
edited Jan 3 at 15:00
Robert Andrzejuk
3,04921528
edited Jan 3 at 15:00
Robert Andrzejuk
3,04921528
3,04921528
asked Jan 3 at 13:50
Fahad KhanFahad Khan
63
asked Jan 3 at 13:50
Fahad KhanFahad Khan
63
asked Jan 3 at 13:50
Fahad KhanFahad Khan
63
63
Can you say a bit more of what you are doing specifically? It could be that your data is good enough with 2 executors and adding executors is just spreading data more.
– Alessandro
Jan 3 at 15:46
I was using mysql database and in that I have 5.7 GB data and I was reading the data via spark. When I was running a query on it via spark it gave me the result in 1.9 minutes(2 executors, 4 cores/executor, executor memory 3gb, partitions 8) but when I increase the number of executors(8 executors, 1 core/executor, 1gb of executor memory, 8 partitions) so I got the query result in 12 minutes. I was wondering why it is so different as it is the same thing I guess.
– Fahad Khan
Jan 4 at 14:38
add a comment |
Can you say a bit more of what you are doing specifically? It could be that your data is good enough with 2 executors and adding executors is just spreading data more.
– Alessandro
Jan 3 at 15:46
I was using mysql database and in that I have 5.7 GB data and I was reading the data via spark. When I was running a query on it via spark it gave me the result in 1.9 minutes(2 executors, 4 cores/executor, executor memory 3gb, partitions 8) but when I increase the number of executors(8 executors, 1 core/executor, 1gb of executor memory, 8 partitions) so I got the query result in 12 minutes. I was wondering why it is so different as it is the same thing I guess.
– Fahad Khan
Jan 4 at 14:38
Can you say a bit more of what you are doing specifically? It could be that your data is good enough with 2 executors and adding executors is just spreading data more.
– Alessandro
Jan 3 at 15:46
Can you say a bit more of what you are doing specifically? It could be that your data is good enough with 2 executors and adding executors is just spreading data more.
– Alessandro
Jan 3 at 15:46
I was using mysql database and in that I have 5.7 GB data and I was reading the data via spark. When I was running a query on it via spark it gave me the result in 1.9 minutes(2 executors, 4 cores/executor, executor memory 3gb, partitions 8) but when I increase the number of executors(8 executors, 1 core/executor, 1gb of executor memory, 8 partitions) so I got the query result in 12 minutes. I was wondering why it is so different as it is the same thing I guess.
– Fahad Khan
Jan 4 at 14:38
I was using mysql database and in that I have 5.7 GB data and I was reading the data via spark. When I was running a query on it via spark it gave me the result in 1.9 minutes(2 executors, 4 cores/executor, executor memory 3gb, partitions 8) but when I increase the number of executors(8 executors, 1 core/executor, 1gb of executor memory, 8 partitions) so I got the query result in 12 minutes. I was wondering why it is so different as it is the same thing I guess.
– Fahad Khan
Jan 4 at 14:38
add a comment |
0
active
oldest
votes
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54023609%2fspark-why-application-slower-with-more-executors%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54023609%2fspark-why-application-slower-with-more-executors%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Can you say a bit more of what you are doing specifically? It could be that your data is good enough with 2 executors and adding executors is just spreading data more.
– Alessandro
Jan 3 at 15:46
I was using mysql database and in that I have 5.7 GB data and I was reading the data via spark. When I was running a query on it via spark it gave me the result in 1.9 minutes(2 executors, 4 cores/executor, executor memory 3gb, partitions 8) but when I increase the number of executors(8 executors, 1 core/executor, 1gb of executor memory, 8 partitions) so I got the query result in 12 minutes. I was wondering why it is so different as it is the same thing I guess.
– Fahad Khan
Jan 4 at 14:38