“IOError: size mismatch in get!” when retrieving files via SFTP

Multi tool use
I have a script which I use to retrieve specific files via SFTP on a regular basis. On occasion, the script will error out with the following output:
Traceback (most recent call last):
File "ETL.py", line 304, in <module>
get_all_files(startdate, enddate, "vma" +
foldernumber + "/logs/", txtype[1] + single_date2 + ".log", txtype[2] +
foldernumber + "\", sftp)
File "ETL.py", line 283, in get_all_files
sftp.get(sftp_dir + filename, local_dir + filename)
File "C:Python27libsite-packagespysftp__init__.py", line 249, in get
self._sftp.get(remotepath, localpath, callback=callback)
File "C:Python27libsite-packagesparamikosftp_client.py", line 806, in get
"size mismatch in get! {} != {}".format(s.st_size, size)
IOError: size mismatch in get! 950272 != 1018742
I have looked through the Paramiko documentation and do not see an explanation for what would trigger this error. Furthermore, the code often works successfully on subsequent tries, or will run successfully for the first few files in the date range and then error out in the middle of downloading all the files I need to retrieve. Other answers on SO say it might be related to the space available on the drive, but I have tried clearing out the destination folder and it hasn't helped. I am trying to download to a network drive/cloud storage if that makes any difference.
Here is the function and code I am using to retrieve the files (via Paramiko):
def get_all_files(start_date, end_date, sftp_dir, filename, local_dir,
sftp_connection):
sftp.get(sftp_dir + filename, local_dir + filename)
with pysftp.Connection('******.com', username='*****', password='******', cnopts=cnopts) as sftp:
get_all_files(startdate, enddate, "vma" + foldernumber + "/logs/", txtype[1] + single_date2 + ".log", txtype[2] + foldernumber + "\", sftp)
I would like all downloadable files to be retrieved without producing this error.
python paramiko pysftp
|
show 2 more comments
I have a script which I use to retrieve specific files via SFTP on a regular basis. On occasion, the script will error out with the following output:
Traceback (most recent call last):
File "ETL.py", line 304, in <module>
get_all_files(startdate, enddate, "vma" +
foldernumber + "/logs/", txtype[1] + single_date2 + ".log", txtype[2] +
foldernumber + "\", sftp)
File "ETL.py", line 283, in get_all_files
sftp.get(sftp_dir + filename, local_dir + filename)
File "C:Python27libsite-packagespysftp__init__.py", line 249, in get
self._sftp.get(remotepath, localpath, callback=callback)
File "C:Python27libsite-packagesparamikosftp_client.py", line 806, in get
"size mismatch in get! {} != {}".format(s.st_size, size)
IOError: size mismatch in get! 950272 != 1018742
I have looked through the Paramiko documentation and do not see an explanation for what would trigger this error. Furthermore, the code often works successfully on subsequent tries, or will run successfully for the first few files in the date range and then error out in the middle of downloading all the files I need to retrieve. Other answers on SO say it might be related to the space available on the drive, but I have tried clearing out the destination folder and it hasn't helped. I am trying to download to a network drive/cloud storage if that makes any difference.
Here is the function and code I am using to retrieve the files (via Paramiko):
def get_all_files(start_date, end_date, sftp_dir, filename, local_dir,
sftp_connection):
sftp.get(sftp_dir + filename, local_dir + filename)
with pysftp.Connection('******.com', username='*****', password='******', cnopts=cnopts) as sftp:
get_all_files(startdate, enddate, "vma" + foldernumber + "/logs/", txtype[1] + single_date2 + ".log", txtype[2] + foldernumber + "\", sftp)
I would like all downloadable files to be retrieved without producing this error.
python paramiko pysftp
Then how come I wouldn't always get the error?
– joshv2
2 days ago
So how exactly large is the file after your Python code finishes?
– Martin Prikryl
2 days ago
@MartinPrikryl It depends on the size of the file I am downloading. The files sizes vary. It doesn't matter to me the size as long as the whole file downloads, which is what I am guessing is the error is trying to alert me to.
– joshv2
2 days ago
I mean if the size is 950272 or 1018742 - in this specific case
– Martin Prikryl
2 days ago
1,018,744 bytes
– joshv2
2 days ago
|
show 2 more comments
I have a script which I use to retrieve specific files via SFTP on a regular basis. On occasion, the script will error out with the following output:
Traceback (most recent call last):
File "ETL.py", line 304, in <module>
get_all_files(startdate, enddate, "vma" +
foldernumber + "/logs/", txtype[1] + single_date2 + ".log", txtype[2] +
foldernumber + "\", sftp)
File "ETL.py", line 283, in get_all_files
sftp.get(sftp_dir + filename, local_dir + filename)
File "C:Python27libsite-packagespysftp__init__.py", line 249, in get
self._sftp.get(remotepath, localpath, callback=callback)
File "C:Python27libsite-packagesparamikosftp_client.py", line 806, in get
"size mismatch in get! {} != {}".format(s.st_size, size)
IOError: size mismatch in get! 950272 != 1018742
I have looked through the Paramiko documentation and do not see an explanation for what would trigger this error. Furthermore, the code often works successfully on subsequent tries, or will run successfully for the first few files in the date range and then error out in the middle of downloading all the files I need to retrieve. Other answers on SO say it might be related to the space available on the drive, but I have tried clearing out the destination folder and it hasn't helped. I am trying to download to a network drive/cloud storage if that makes any difference.
Here is the function and code I am using to retrieve the files (via Paramiko):
def get_all_files(start_date, end_date, sftp_dir, filename, local_dir,
sftp_connection):
sftp.get(sftp_dir + filename, local_dir + filename)
with pysftp.Connection('******.com', username='*****', password='******', cnopts=cnopts) as sftp:
get_all_files(startdate, enddate, "vma" + foldernumber + "/logs/", txtype[1] + single_date2 + ".log", txtype[2] + foldernumber + "\", sftp)
I would like all downloadable files to be retrieved without producing this error.
python paramiko pysftp
I have a script which I use to retrieve specific files via SFTP on a regular basis. On occasion, the script will error out with the following output:
Traceback (most recent call last):
File "ETL.py", line 304, in <module>
get_all_files(startdate, enddate, "vma" +
foldernumber + "/logs/", txtype[1] + single_date2 + ".log", txtype[2] +
foldernumber + "\", sftp)
File "ETL.py", line 283, in get_all_files
sftp.get(sftp_dir + filename, local_dir + filename)
File "C:Python27libsite-packagespysftp__init__.py", line 249, in get
self._sftp.get(remotepath, localpath, callback=callback)
File "C:Python27libsite-packagesparamikosftp_client.py", line 806, in get
"size mismatch in get! {} != {}".format(s.st_size, size)
IOError: size mismatch in get! 950272 != 1018742
I have looked through the Paramiko documentation and do not see an explanation for what would trigger this error. Furthermore, the code often works successfully on subsequent tries, or will run successfully for the first few files in the date range and then error out in the middle of downloading all the files I need to retrieve. Other answers on SO say it might be related to the space available on the drive, but I have tried clearing out the destination folder and it hasn't helped. I am trying to download to a network drive/cloud storage if that makes any difference.
Here is the function and code I am using to retrieve the files (via Paramiko):
def get_all_files(start_date, end_date, sftp_dir, filename, local_dir,
sftp_connection):
sftp.get(sftp_dir + filename, local_dir + filename)
with pysftp.Connection('******.com', username='*****', password='******', cnopts=cnopts) as sftp:
get_all_files(startdate, enddate, "vma" + foldernumber + "/logs/", txtype[1] + single_date2 + ".log", txtype[2] + foldernumber + "\", sftp)
I would like all downloadable files to be retrieved without producing this error.
python paramiko pysftp
python paramiko pysftp
asked Dec 27 at 13:05
joshv2
4519
4519
Then how come I wouldn't always get the error?
– joshv2
2 days ago
So how exactly large is the file after your Python code finishes?
– Martin Prikryl
2 days ago
@MartinPrikryl It depends on the size of the file I am downloading. The files sizes vary. It doesn't matter to me the size as long as the whole file downloads, which is what I am guessing is the error is trying to alert me to.
– joshv2
2 days ago
I mean if the size is 950272 or 1018742 - in this specific case
– Martin Prikryl
2 days ago
1,018,744 bytes
– joshv2
2 days ago
|
show 2 more comments
Then how come I wouldn't always get the error?
– joshv2
2 days ago
So how exactly large is the file after your Python code finishes?
– Martin Prikryl
2 days ago
@MartinPrikryl It depends on the size of the file I am downloading. The files sizes vary. It doesn't matter to me the size as long as the whole file downloads, which is what I am guessing is the error is trying to alert me to.
– joshv2
2 days ago
I mean if the size is 950272 or 1018742 - in this specific case
– Martin Prikryl
2 days ago
1,018,744 bytes
– joshv2
2 days ago
Then how come I wouldn't always get the error?
– joshv2
2 days ago
Then how come I wouldn't always get the error?
– joshv2
2 days ago
So how exactly large is the file after your Python code finishes?
– Martin Prikryl
2 days ago
So how exactly large is the file after your Python code finishes?
– Martin Prikryl
2 days ago
@MartinPrikryl It depends on the size of the file I am downloading. The files sizes vary. It doesn't matter to me the size as long as the whole file downloads, which is what I am guessing is the error is trying to alert me to.
– joshv2
2 days ago
@MartinPrikryl It depends on the size of the file I am downloading. The files sizes vary. It doesn't matter to me the size as long as the whole file downloads, which is what I am guessing is the error is trying to alert me to.
– joshv2
2 days ago
I mean if the size is 950272 or 1018742 - in this specific case
– Martin Prikryl
2 days ago
I mean if the size is 950272 or 1018742 - in this specific case
– Martin Prikryl
2 days ago
1,018,744 bytes
– joshv2
2 days ago
1,018,744 bytes
– joshv2
2 days ago
|
show 2 more comments
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53945594%2fioerror-size-mismatch-in-get-when-retrieving-files-via-sftp%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53945594%2fioerror-size-mismatch-in-get-when-retrieving-files-via-sftp%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
LZloVFsauCkOP8bt4ne0
Then how come I wouldn't always get the error?
– joshv2
2 days ago
So how exactly large is the file after your Python code finishes?
– Martin Prikryl
2 days ago
@MartinPrikryl It depends on the size of the file I am downloading. The files sizes vary. It doesn't matter to me the size as long as the whole file downloads, which is what I am guessing is the error is trying to alert me to.
– joshv2
2 days ago
I mean if the size is 950272 or 1018742 - in this specific case
– Martin Prikryl
2 days ago
1,018,744 bytes
– joshv2
2 days ago