best way to parallelise for loop on contiguous array
I have a function f() that loops over a 2 dimensional array and applies another function g() to each element of the array, and stores the results. The evaluations of g() from one array element to another are completely independent. I would like to speed this up, since calculating g() is quick, but doing it for all elements of a large array is quite slow..
I am not familiar with multiprocessing in python, can anyone provide the best approach to dealing with this kind of problem? The code looks something like this:
def g(x):
'''some function that is dependent on other python packages'''
return other_func(x)
def f(arr):
N = arr.shape[0]
M = arr.shape[1]
results = np.zeros_like(arr)
for i in range(N):
for j in range(M):
results[i,j] = g(arr[i,j])
return results
*update : We can assume that M=N=100 and that the evaluation of g() takes a few seconds, t<10, so the total running time currently is t * 100^2. My machine is a macbook pro with 2.6Ghz i7 and 16gb ram
python-3.x parallel-processing
asked Jan 2 at 4:04
dimebucker91dimebucker91
463316
add a comment |
I have a function f() that loops over a 2 dimensional array and applies another function g() to each element of the array, and stores the results. The evaluations of g() from one array element to another are completely independent. I would like to speed this up, since calculating g() is quick, but doing it for all elements of a large array is quite slow..
I am not familiar with multiprocessing in python, can anyone provide the best approach to dealing with this kind of problem? The code looks something like this:
def g(x):
'''some function that is dependent on other python packages'''
return other_func(x)
def f(arr):
N = arr.shape[0]
M = arr.shape[1]
results = np.zeros_like(arr)
for i in range(N):
for j in range(M):
results[i,j] = g(arr[i,j])
return results
*update : We can assume that M=N=100 and that the evaluation of g() takes a few seconds, t<10, so the total running time currently is t * 100^2. My machine is a macbook pro with 2.6Ghz i7 and 16gb ram
python-3.x parallel-processing
asked Jan 2 at 4:04
dimebucker91dimebucker91
463316
Please clarify what you mean by "slow" and give some indication of the sizes of the array (M and N). Also, please say what machine you are running on.
– Mark Setchell
Jan 3 at 15:33
@MarkSetchell I've edited to reflect this
– dimebucker91
Jan 4 at 5:21
add a comment |
I have a function f() that loops over a 2 dimensional array and applies another function g() to each element of the array, and stores the results. The evaluations of g() from one array element to another are completely independent. I would like to speed this up, since calculating g() is quick, but doing it for all elements of a large array is quite slow..
I am not familiar with multiprocessing in python, can anyone provide the best approach to dealing with this kind of problem? The code looks something like this:
def g(x):
'''some function that is dependent on other python packages'''
return other_func(x)
def f(arr):
N = arr.shape[0]
M = arr.shape[1]
results = np.zeros_like(arr)
for i in range(N):
for j in range(M):
results[i,j] = g(arr[i,j])
return results
*update : We can assume that M=N=100 and that the evaluation of g() takes a few seconds, t<10, so the total running time currently is t * 100^2. My machine is a macbook pro with 2.6Ghz i7 and 16gb ram
python-3.x parallel-processing
asked Jan 2 at 4:04
dimebucker91dimebucker91
463316
I have a function f() that loops over a 2 dimensional array and applies another function g() to each element of the array, and stores the results. The evaluations of g() from one array element to another are completely independent. I would like to speed this up, since calculating g() is quick, but doing it for all elements of a large array is quite slow..
I am not familiar with multiprocessing in python, can anyone provide the best approach to dealing with this kind of problem? The code looks something like this:
def g(x):
'''some function that is dependent on other python packages'''
return other_func(x)
def f(arr):
N = arr.shape[0]
M = arr.shape[1]
results = np.zeros_like(arr)
for i in range(N):
for j in range(M):
results[i,j] = g(arr[i,j])
return results
*update : We can assume that M=N=100 and that the evaluation of g() takes a few seconds, t<10, so the total running time currently is t * 100^2. My machine is a macbook pro with 2.6Ghz i7 and 16gb ram
python-3.x parallel-processing
python-3.x parallel-processing
asked Jan 2 at 4:04
dimebucker91dimebucker91
463316
asked Jan 2 at 4:04
dimebucker91dimebucker91
463316
edited Jan 4 at 5:23
dimebucker91
asked Jan 2 at 4:04
dimebucker91dimebucker91
463316
asked Jan 2 at 4:04
dimebucker91dimebucker91
463316
asked Jan 2 at 4:04
dimebucker91dimebucker91
463316
463316
Please clarify what you mean by "slow" and give some indication of the sizes of the array (M and N). Also, please say what machine you are running on.
– Mark Setchell
Jan 3 at 15:33
@MarkSetchell I've edited to reflect this
– dimebucker91
Jan 4 at 5:21
add a comment |
Please clarify what you mean by "slow" and give some indication of the sizes of the array (M and N). Also, please say what machine you are running on.
– Mark Setchell
Jan 3 at 15:33
@MarkSetchell I've edited to reflect this
– dimebucker91
Jan 4 at 5:21
Please clarify what you mean by "slow" and give some indication of the sizes of the array (M and N). Also, please say what machine you are running on.
– Mark Setchell
Jan 3 at 15:33
Please clarify what you mean by "slow" and give some indication of the sizes of the array (M and N). Also, please say what machine you are running on.
– Mark Setchell
Jan 3 at 15:33
@MarkSetchell I've edited to reflect this
– dimebucker91
Jan 4 at 5:21
@MarkSetchell I've edited to reflect this
– dimebucker91
Jan 4 at 5:21
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54001063%2fbest-way-to-parallelise-for-loop-on-contiguous-array%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54001063%2fbest-way-to-parallelise-for-loop-on-contiguous-array%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Please clarify what you mean by "slow" and give some indication of the sizes of the array (M and N). Also, please say what machine you are running on.
– Mark Setchell
Jan 3 at 15:33
@MarkSetchell I've edited to reflect this
– dimebucker91
Jan 4 at 5:21