subprocess vs asyncio for asynchronous postgresql queries
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I want to perform a number of operations on a postgresql database. These operations carry out a select on a table and then insert the resulting rows into a new table which has primary keys, ignoring rows which violate the primary key constraints. There are a large number of large tables in the database to be processed, and it seems that this sort of task should be run asynchronously.
It strikes me that one way to go about this would be to use the subprocess module in Python to run bash scripts which perform these operations, using something like subprocess.Popen. I can open many terminal sessions and execute queries in parallel and to my understanding this approach is imitating this.
To borrow an example from here:
from subprocess import Popen, PIPE
import glob
f_list = glob.glob('./*bz2')
cmds_list = [['./bunzip2_file.py', file_name] for file_name in f_list]
procs_list = [Popen(cmd, stdout=PIPE, stderr=PIPE) for cmd in cmds_list]
for proc in procs_list:
proc.wait()
My questions are:
Are there any obvious issues with calling many postgres queries using
subprocess?Under what circumstances might I instead consider using
asyncio? Does it provide any advantages to the method discussed above?
python postgresql subprocess python-asyncio
asked Jan 4 at 16:56
BHCBHC
16911
add a comment |
I want to perform a number of operations on a postgresql database. These operations carry out a select on a table and then insert the resulting rows into a new table which has primary keys, ignoring rows which violate the primary key constraints. There are a large number of large tables in the database to be processed, and it seems that this sort of task should be run asynchronously.
It strikes me that one way to go about this would be to use the subprocess module in Python to run bash scripts which perform these operations, using something like subprocess.Popen. I can open many terminal sessions and execute queries in parallel and to my understanding this approach is imitating this.
To borrow an example from here:
from subprocess import Popen, PIPE
import glob
f_list = glob.glob('./*bz2')
cmds_list = [['./bunzip2_file.py', file_name] for file_name in f_list]
procs_list = [Popen(cmd, stdout=PIPE, stderr=PIPE) for cmd in cmds_list]
for proc in procs_list:
proc.wait()
My questions are:
Are there any obvious issues with calling many postgres queries using
subprocess?Under what circumstances might I instead consider using
asyncio? Does it provide any advantages to the method discussed above?
python postgresql subprocess python-asyncio
asked Jan 4 at 16:56
BHCBHC
16911
add a comment |
I want to perform a number of operations on a postgresql database. These operations carry out a select on a table and then insert the resulting rows into a new table which has primary keys, ignoring rows which violate the primary key constraints. There are a large number of large tables in the database to be processed, and it seems that this sort of task should be run asynchronously.
It strikes me that one way to go about this would be to use the subprocess module in Python to run bash scripts which perform these operations, using something like subprocess.Popen. I can open many terminal sessions and execute queries in parallel and to my understanding this approach is imitating this.
To borrow an example from here:
from subprocess import Popen, PIPE
import glob
f_list = glob.glob('./*bz2')
cmds_list = [['./bunzip2_file.py', file_name] for file_name in f_list]
procs_list = [Popen(cmd, stdout=PIPE, stderr=PIPE) for cmd in cmds_list]
for proc in procs_list:
proc.wait()
My questions are:
Are there any obvious issues with calling many postgres queries using
subprocess?Under what circumstances might I instead consider using
asyncio? Does it provide any advantages to the method discussed above?
python postgresql subprocess python-asyncio
asked Jan 4 at 16:56
BHCBHC
16911
I want to perform a number of operations on a postgresql database. These operations carry out a select on a table and then insert the resulting rows into a new table which has primary keys, ignoring rows which violate the primary key constraints. There are a large number of large tables in the database to be processed, and it seems that this sort of task should be run asynchronously.
It strikes me that one way to go about this would be to use the subprocess module in Python to run bash scripts which perform these operations, using something like subprocess.Popen. I can open many terminal sessions and execute queries in parallel and to my understanding this approach is imitating this.
To borrow an example from here:
from subprocess import Popen, PIPE
import glob
f_list = glob.glob('./*bz2')
cmds_list = [['./bunzip2_file.py', file_name] for file_name in f_list]
procs_list = [Popen(cmd, stdout=PIPE, stderr=PIPE) for cmd in cmds_list]
for proc in procs_list:
proc.wait()
My questions are:
Are there any obvious issues with calling many postgres queries using
subprocess?Under what circumstances might I instead consider using
asyncio? Does it provide any advantages to the method discussed above?
python postgresql subprocess python-asyncio
python postgresql subprocess python-asyncio
asked Jan 4 at 16:56
BHCBHC
16911
asked Jan 4 at 16:56
BHCBHC
16911
edited Jan 4 at 17:01
BHC
asked Jan 4 at 16:56
BHCBHC
16911
asked Jan 4 at 16:56
BHCBHC
16911
asked Jan 4 at 16:56
BHCBHC
16911
16911
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Note, that asyncio itself is about controlling execution flow in the first place. It means, for example, you can flexibly manage subprocesses using asyncio. So your question is actually about using processes vs. PostgreSQL async driver.
First of all you probably don't need processes: if your bash scripts don't contain much calculations you can use threads, they're cheaper.
When it come to asyncio vs. threads both solve main performance bottleneck - network I/O. You probably won't see any performance difference unless you're spawning thousands of threads (see this question and answer for an example).
answered Jan 6 at 15:41
Mikhail GerasimovMikhail Gerasimov
15.3k44172
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54043122%2fsubprocess-vs-asyncio-for-asynchronous-postgresql-queries%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Note, that asyncio itself is about controlling execution flow in the first place. It means, for example, you can flexibly manage subprocesses using asyncio. So your question is actually about using processes vs. PostgreSQL async driver.
First of all you probably don't need processes: if your bash scripts don't contain much calculations you can use threads, they're cheaper.
When it come to asyncio vs. threads both solve main performance bottleneck - network I/O. You probably won't see any performance difference unless you're spawning thousands of threads (see this question and answer for an example).
answered Jan 6 at 15:41
Mikhail GerasimovMikhail Gerasimov
15.3k44172
add a comment |
Note, that asyncio itself is about controlling execution flow in the first place. It means, for example, you can flexibly manage subprocesses using asyncio. So your question is actually about using processes vs. PostgreSQL async driver.
First of all you probably don't need processes: if your bash scripts don't contain much calculations you can use threads, they're cheaper.
When it come to asyncio vs. threads both solve main performance bottleneck - network I/O. You probably won't see any performance difference unless you're spawning thousands of threads (see this question and answer for an example).
answered Jan 6 at 15:41
Mikhail GerasimovMikhail Gerasimov
15.3k44172
add a comment |
Note, that asyncio itself is about controlling execution flow in the first place. It means, for example, you can flexibly manage subprocesses using asyncio. So your question is actually about using processes vs. PostgreSQL async driver.
First of all you probably don't need processes: if your bash scripts don't contain much calculations you can use threads, they're cheaper.
When it come to asyncio vs. threads both solve main performance bottleneck - network I/O. You probably won't see any performance difference unless you're spawning thousands of threads (see this question and answer for an example).
answered Jan 6 at 15:41
Mikhail GerasimovMikhail Gerasimov
15.3k44172
Note, that asyncio itself is about controlling execution flow in the first place. It means, for example, you can flexibly manage subprocesses using asyncio. So your question is actually about using processes vs. PostgreSQL async driver.
First of all you probably don't need processes: if your bash scripts don't contain much calculations you can use threads, they're cheaper.
When it come to asyncio vs. threads both solve main performance bottleneck - network I/O. You probably won't see any performance difference unless you're spawning thousands of threads (see this question and answer for an example).
answered Jan 6 at 15:41
Mikhail GerasimovMikhail Gerasimov
15.3k44172
answered Jan 6 at 15:41
Mikhail GerasimovMikhail Gerasimov
15.3k44172
answered Jan 6 at 15:41
Mikhail GerasimovMikhail Gerasimov
15.3k44172
answered Jan 6 at 15:41
Mikhail GerasimovMikhail Gerasimov
15.3k44172
15.3k44172
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54043122%2fsubprocess-vs-asyncio-for-asynchronous-postgresql-queries%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
