How to save variables before killing a (too long) running python script?
I've been running a python script using:
- python
- Anacondaon
- linux server (4.15.0-36-generic #39~16.04.1-Ubuntu)
For two weeks now and I think it will take more than a month to finish. This is already too long and so I would like to kill the process but before that I would like to save some lists that have been constructed during the last two weeks.
So far I tried:
1) using pyrasite-shell with one the PID associated with my python script (I've more than 20 PIDs associated with my script according to htop) but it does not work (no shell opens).
2) I also tried to find some kernel address using ps -aux | grep python3 and then connect to this kernel via something like ipython console --existing d3d55f0d-52f1-4660-a43b-1fedb698452f.json and then look for local variables with locals()in the prompt shell but this did not work (none of my lists appear in the list of local variables).
Ideally, I would like to connect the python kernel running my code and save all the current lists via pickle. Or any other idea to retrieve the lists would be great.
python-3.x
edited Jan 3 at 16:15
jalazbe
511318
asked Jan 3 at 14:44
guillaume lecueguillaume lecue
84
add a comment |
I've been running a python script using:
- python
- Anacondaon
- linux server (4.15.0-36-generic #39~16.04.1-Ubuntu)
For two weeks now and I think it will take more than a month to finish. This is already too long and so I would like to kill the process but before that I would like to save some lists that have been constructed during the last two weeks.
So far I tried:
1) using pyrasite-shell with one the PID associated with my python script (I've more than 20 PIDs associated with my script according to htop) but it does not work (no shell opens).
2) I also tried to find some kernel address using ps -aux | grep python3 and then connect to this kernel via something like ipython console --existing d3d55f0d-52f1-4660-a43b-1fedb698452f.json and then look for local variables with locals()in the prompt shell but this did not work (none of my lists appear in the list of local variables).
Ideally, I would like to connect the python kernel running my code and save all the current lists via pickle. Or any other idea to retrieve the lists would be great.
python-3.x
edited Jan 3 at 16:15
jalazbe
511318
asked Jan 3 at 14:44
guillaume lecueguillaume lecue
84
how about a finally statement with try/catch?
– SRT HellKitty
Jan 3 at 16:20
2
Colloquially you setup a logging system like this before you run your script...have you triedrconsole?
– Matt Messersmith
Jan 3 at 16:20
No, I've not triedrconsole(and I did not find any information about it). What is it? do you have some link that may explain how that works? Thanks.
– guillaume lecue
Jan 4 at 10:18
add a comment |
I've been running a python script using:
- python
- Anacondaon
- linux server (4.15.0-36-generic #39~16.04.1-Ubuntu)
For two weeks now and I think it will take more than a month to finish. This is already too long and so I would like to kill the process but before that I would like to save some lists that have been constructed during the last two weeks.
So far I tried:
1) using pyrasite-shell with one the PID associated with my python script (I've more than 20 PIDs associated with my script according to htop) but it does not work (no shell opens).
2) I also tried to find some kernel address using ps -aux | grep python3 and then connect to this kernel via something like ipython console --existing d3d55f0d-52f1-4660-a43b-1fedb698452f.json and then look for local variables with locals()in the prompt shell but this did not work (none of my lists appear in the list of local variables).
Ideally, I would like to connect the python kernel running my code and save all the current lists via pickle. Or any other idea to retrieve the lists would be great.
python-3.x
edited Jan 3 at 16:15
jalazbe
511318
asked Jan 3 at 14:44
guillaume lecueguillaume lecue
84
I've been running a python script using:
- python
- Anacondaon
- linux server (4.15.0-36-generic #39~16.04.1-Ubuntu)
For two weeks now and I think it will take more than a month to finish. This is already too long and so I would like to kill the process but before that I would like to save some lists that have been constructed during the last two weeks.
So far I tried:
1) using pyrasite-shell with one the PID associated with my python script (I've more than 20 PIDs associated with my script according to htop) but it does not work (no shell opens).
2) I also tried to find some kernel address using ps -aux | grep python3 and then connect to this kernel via something like ipython console --existing d3d55f0d-52f1-4660-a43b-1fedb698452f.json and then look for local variables with locals()in the prompt shell but this did not work (none of my lists appear in the list of local variables).
Ideally, I would like to connect the python kernel running my code and save all the current lists via pickle. Or any other idea to retrieve the lists would be great.
python-3.x
python-3.x
edited Jan 3 at 16:15
jalazbe
511318
asked Jan 3 at 14:44
guillaume lecueguillaume lecue
84
edited Jan 3 at 16:15
jalazbe
511318
asked Jan 3 at 14:44
guillaume lecueguillaume lecue
84
edited Jan 3 at 16:15
jalazbe
511318
edited Jan 3 at 16:15
jalazbe
511318
edited Jan 3 at 16:15
jalazbe
511318
511318
asked Jan 3 at 14:44
guillaume lecueguillaume lecue
84
asked Jan 3 at 14:44
guillaume lecueguillaume lecue
84
asked Jan 3 at 14:44
guillaume lecueguillaume lecue
84
84
how about a finally statement with try/catch?
– SRT HellKitty
Jan 3 at 16:20
2
Colloquially you setup a logging system like this before you run your script...have you triedrconsole?
– Matt Messersmith
Jan 3 at 16:20
No, I've not triedrconsole(and I did not find any information about it). What is it? do you have some link that may explain how that works? Thanks.
– guillaume lecue
Jan 4 at 10:18
add a comment |
how about a finally statement with try/catch?
– SRT HellKitty
Jan 3 at 16:20
2
Colloquially you setup a logging system like this before you run your script...have you triedrconsole?
– Matt Messersmith
Jan 3 at 16:20
No, I've not triedrconsole(and I did not find any information about it). What is it? do you have some link that may explain how that works? Thanks.
– guillaume lecue
Jan 4 at 10:18
how about a finally statement with try/catch?
– SRT HellKitty
Jan 3 at 16:20
how about a finally statement with try/catch?
– SRT HellKitty
Jan 3 at 16:20
2
2
Colloquially you setup a logging system like this before you run your script...have you tried
rconsole?– Matt Messersmith
Jan 3 at 16:20
Colloquially you setup a logging system like this before you run your script...have you tried
rconsole?– Matt Messersmith
Jan 3 at 16:20
No, I've not tried
rconsole (and I did not find any information about it). What is it? do you have some link that may explain how that works? Thanks.– guillaume lecue
Jan 4 at 10:18
No, I've not tried
rconsole (and I did not find any information about it). What is it? do you have some link that may explain how that works? Thanks.– guillaume lecue
Jan 4 at 10:18
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54024538%2fhow-to-save-variables-before-killing-a-too-long-running-python-script%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54024538%2fhow-to-save-variables-before-killing-a-too-long-running-python-script%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
how about a finally statement with try/catch?
– SRT HellKitty
Jan 3 at 16:20
2
Colloquially you setup a logging system like this before you run your script...have you tried
rconsole?– Matt Messersmith
Jan 3 at 16:20
No, I've not tried
rconsole(and I did not find any information about it). What is it? do you have some link that may explain how that works? Thanks.– guillaume lecue
Jan 4 at 10:18