Reduce Python shelve size
I'm using the
shelve module with gdbm to store Python objects. I understand that shelve uses pickle to store the objects. Unfortunately, the sizes of my shelves are too large. I found this solution to bzip2 or gzip individual pickles. My question is: is there a way to make shelve compress all its pickles?
python pickle shelve
asked Dec 31 '18 at 18:43
BotondBotond
4691922
add a comment |
I'm using the
shelve module with gdbm to store Python objects. I understand that shelve uses pickle to store the objects. Unfortunately, the sizes of my shelves are too large. I found this solution to bzip2 or gzip individual pickles. My question is: is there a way to make shelve compress all its pickles?
python pickle shelve
asked Dec 31 '18 at 18:43
BotondBotond
4691922
if all your files are in the same directory just dogzip *from the CLI
– aws_apprentice
Dec 31 '18 at 19:08
@aws_apprentice the problem is that I would need to access shelve entries individually.
– Botond
Dec 31 '18 at 19:09
so? it will zip each one separately
– aws_apprentice
Dec 31 '18 at 19:19
Shelf is a single file containing many pickled objects.
– Botond
Dec 31 '18 at 19:30
add a comment |
I'm using the
shelve module with gdbm to store Python objects. I understand that shelve uses pickle to store the objects. Unfortunately, the sizes of my shelves are too large. I found this solution to bzip2 or gzip individual pickles. My question is: is there a way to make shelve compress all its pickles?
python pickle shelve
asked Dec 31 '18 at 18:43
BotondBotond
4691922
I'm using the
shelve module with gdbm to store Python objects. I understand that shelve uses pickle to store the objects. Unfortunately, the sizes of my shelves are too large. I found this solution to bzip2 or gzip individual pickles. My question is: is there a way to make shelve compress all its pickles?
python pickle shelve
python pickle shelve
asked Dec 31 '18 at 18:43
BotondBotond
4691922
asked Dec 31 '18 at 18:43
BotondBotond
4691922
asked Dec 31 '18 at 18:43
BotondBotond
4691922
asked Dec 31 '18 at 18:43
BotondBotond
4691922
asked Dec 31 '18 at 18:43
BotondBotond
4691922
4691922
if all your files are in the same directory just dogzip *from the CLI
– aws_apprentice
Dec 31 '18 at 19:08
@aws_apprentice the problem is that I would need to access shelve entries individually.
– Botond
Dec 31 '18 at 19:09
so? it will zip each one separately
– aws_apprentice
Dec 31 '18 at 19:19
Shelf is a single file containing many pickled objects.
– Botond
Dec 31 '18 at 19:30
add a comment |
if all your files are in the same directory just dogzip *from the CLI
– aws_apprentice
Dec 31 '18 at 19:08
@aws_apprentice the problem is that I would need to access shelve entries individually.
– Botond
Dec 31 '18 at 19:09
so? it will zip each one separately
– aws_apprentice
Dec 31 '18 at 19:19
Shelf is a single file containing many pickled objects.
– Botond
Dec 31 '18 at 19:30
if all your files are in the same directory just do
gzip * from the CLI– aws_apprentice
Dec 31 '18 at 19:08
if all your files are in the same directory just do
gzip * from the CLI– aws_apprentice
Dec 31 '18 at 19:08
@aws_apprentice the problem is that I would need to access shelve entries individually.
– Botond
Dec 31 '18 at 19:09
@aws_apprentice the problem is that I would need to access shelve entries individually.
– Botond
Dec 31 '18 at 19:09
so? it will zip each one separately
– aws_apprentice
Dec 31 '18 at 19:19
so? it will zip each one separately
– aws_apprentice
Dec 31 '18 at 19:19
Shelf is a single file containing many pickled objects.
– Botond
Dec 31 '18 at 19:30
Shelf is a single file containing many pickled objects.
– Botond
Dec 31 '18 at 19:30
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53990501%2freduce-python-shelve-size%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53990501%2freduce-python-shelve-size%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
if all your files are in the same directory just do
gzip *from the CLI– aws_apprentice
Dec 31 '18 at 19:08
@aws_apprentice the problem is that I would need to access shelve entries individually.
– Botond
Dec 31 '18 at 19:09
so? it will zip each one separately
– aws_apprentice
Dec 31 '18 at 19:19
Shelf is a single file containing many pickled objects.
– Botond
Dec 31 '18 at 19:30