How to write to yaml file in human readable format?
I have some data types I need to write:
a. A list of numpy arrays, e.g. [ndarray, ndarray, ndarray] of different sizes.
b. Any arbitrary numpy array, e.g. np.zeros((5,6)), np.randn((76,2)) and so on.
c. Any other future datatype that hasn't occurred to me yet.
Requirements:
I need a single function to be able to save all those data types, with no specific handling, and with future compatibility for type c stated above.
I also need the output file dump in human readable format.
So far, I was only able to achieve requirement 1 with either YAML or pickle, both of which with binary files, i.e. not human readable.
@staticmethod
def _read_with_yaml(path):
with open(path, 'r') as stream:
return yaml.load(stream)
@staticmethod
def _write_with_yaml(path, obj):
with io.open(path, 'w+', encoding='utf8') as outfile:
yaml.dump(obj, outfile, default_flow_style=False, allow_unicode=True)
This example code outputs non-human-readable files, but works for the data types I have.
Is there a way to achieve both requirements?
python file serialization yaml pickle
asked Dec 31 '18 at 12:33
GulzarGulzar
8631920
add a comment |
I have some data types I need to write:
a. A list of numpy arrays, e.g. [ndarray, ndarray, ndarray] of different sizes.
b. Any arbitrary numpy array, e.g. np.zeros((5,6)), np.randn((76,2)) and so on.
c. Any other future datatype that hasn't occurred to me yet.
Requirements:
I need a single function to be able to save all those data types, with no specific handling, and with future compatibility for type c stated above.
I also need the output file dump in human readable format.
So far, I was only able to achieve requirement 1 with either YAML or pickle, both of which with binary files, i.e. not human readable.
@staticmethod
def _read_with_yaml(path):
with open(path, 'r') as stream:
return yaml.load(stream)
@staticmethod
def _write_with_yaml(path, obj):
with io.open(path, 'w+', encoding='utf8') as outfile:
yaml.dump(obj, outfile, default_flow_style=False, allow_unicode=True)
This example code outputs non-human-readable files, but works for the data types I have.
Is there a way to achieve both requirements?
python file serialization yaml pickle
asked Dec 31 '18 at 12:33
GulzarGulzar
8631920
add a comment |
I have some data types I need to write:
a. A list of numpy arrays, e.g. [ndarray, ndarray, ndarray] of different sizes.
b. Any arbitrary numpy array, e.g. np.zeros((5,6)), np.randn((76,2)) and so on.
c. Any other future datatype that hasn't occurred to me yet.
Requirements:
I need a single function to be able to save all those data types, with no specific handling, and with future compatibility for type c stated above.
I also need the output file dump in human readable format.
So far, I was only able to achieve requirement 1 with either YAML or pickle, both of which with binary files, i.e. not human readable.
@staticmethod
def _read_with_yaml(path):
with open(path, 'r') as stream:
return yaml.load(stream)
@staticmethod
def _write_with_yaml(path, obj):
with io.open(path, 'w+', encoding='utf8') as outfile:
yaml.dump(obj, outfile, default_flow_style=False, allow_unicode=True)
This example code outputs non-human-readable files, but works for the data types I have.
Is there a way to achieve both requirements?
python file serialization yaml pickle
asked Dec 31 '18 at 12:33
GulzarGulzar
8631920
I have some data types I need to write:
a. A list of numpy arrays, e.g. [ndarray, ndarray, ndarray] of different sizes.
b. Any arbitrary numpy array, e.g. np.zeros((5,6)), np.randn((76,2)) and so on.
c. Any other future datatype that hasn't occurred to me yet.
Requirements:
I need a single function to be able to save all those data types, with no specific handling, and with future compatibility for type c stated above.
I also need the output file dump in human readable format.
So far, I was only able to achieve requirement 1 with either YAML or pickle, both of which with binary files, i.e. not human readable.
@staticmethod
def _read_with_yaml(path):
with open(path, 'r') as stream:
return yaml.load(stream)
@staticmethod
def _write_with_yaml(path, obj):
with io.open(path, 'w+', encoding='utf8') as outfile:
yaml.dump(obj, outfile, default_flow_style=False, allow_unicode=True)
This example code outputs non-human-readable files, but works for the data types I have.
Is there a way to achieve both requirements?
python file serialization yaml pickle
python file serialization yaml pickle
asked Dec 31 '18 at 12:33
GulzarGulzar
8631920
asked Dec 31 '18 at 12:33
GulzarGulzar
8631920
edited Jan 1 at 18:27
Gulzar
asked Dec 31 '18 at 12:33
GulzarGulzar
8631920
asked Dec 31 '18 at 12:33
GulzarGulzar
8631920
asked Dec 31 '18 at 12:33
GulzarGulzar
8631920
8631920
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
No, your requirements cannot be satisfied.
You already have one function yaml.dump() that saves all those data types. As you noticed it doesn't do so in a very readable way for numpy data structures. This is caused by numpy not having dumping routines for their special data structure, instead falling back to the, not-so-readable, default !python.... tagged dump of the datastructure. Now you (or the YAML or Numpy package maintainers) can provide, special routines for those objects that dump in a more readable format, so that could be covered. You can make the representer in your YAML library more intelligent and get more readable output for Numpy datastructures without touching Numpy classes.
But you want this for all future datatypes, and IMO a variation of Gödel's theorem applies: even if the YAML library is extended so that it covers all known cases and dumps them in a readable way, there will always be new datastructures, especially in C based extensions (like Numpy), that cannot be represented in a readable way without extra work.
So because of your
Any other future unknown datatype that hasn't occurred to me yet.
premise, this is not just a lot of difficult work, but impossible.
answered Dec 31 '18 at 15:59
AnthonAnthon
29.9k1693147
Are you sure about the strict impossibility? In the end, any data structure would be comprised of other data structures, that, recursively, in the end would be some primitives. I assumed some library implemented that in some tree format, but why would it be impossible?
– Gulzar
Dec 31 '18 at 16:09
Also, what about without the "any future" requirement? Is there some library that handles the other use cases internally?
– Gulzar
Dec 31 '18 at 16:15
There is nothing in ruamel.yaml nor in PyYAML (and I don't think NumPy works with the syck library). In principle a third party library could provide better readable representations, but I don't think any library does. I once looked at doing that for ruamel.yaml, but I would have had to dig in too deeply and I am not a NumPy user, so the motivation for that is low.. You should be able monkeypatch those NumPy classes that you are using. If you need help with that (forruamel.yaml) post another question with the full program that dumps, what the output looks like, and what output you want
– Anthon
Dec 31 '18 at 18:25
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53987582%2fhow-to-write-to-yaml-file-in-human-readable-format%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
No, your requirements cannot be satisfied.
You already have one function yaml.dump() that saves all those data types. As you noticed it doesn't do so in a very readable way for numpy data structures. This is caused by numpy not having dumping routines for their special data structure, instead falling back to the, not-so-readable, default !python.... tagged dump of the datastructure. Now you (or the YAML or Numpy package maintainers) can provide, special routines for those objects that dump in a more readable format, so that could be covered. You can make the representer in your YAML library more intelligent and get more readable output for Numpy datastructures without touching Numpy classes.
But you want this for all future datatypes, and IMO a variation of Gödel's theorem applies: even if the YAML library is extended so that it covers all known cases and dumps them in a readable way, there will always be new datastructures, especially in C based extensions (like Numpy), that cannot be represented in a readable way without extra work.
So because of your
Any other future unknown datatype that hasn't occurred to me yet.
premise, this is not just a lot of difficult work, but impossible.
answered Dec 31 '18 at 15:59
AnthonAnthon
29.9k1693147
Are you sure about the strict impossibility? In the end, any data structure would be comprised of other data structures, that, recursively, in the end would be some primitives. I assumed some library implemented that in some tree format, but why would it be impossible?
– Gulzar
Dec 31 '18 at 16:09
Also, what about without the "any future" requirement? Is there some library that handles the other use cases internally?
– Gulzar
Dec 31 '18 at 16:15
There is nothing in ruamel.yaml nor in PyYAML (and I don't think NumPy works with the syck library). In principle a third party library could provide better readable representations, but I don't think any library does. I once looked at doing that for ruamel.yaml, but I would have had to dig in too deeply and I am not a NumPy user, so the motivation for that is low.. You should be able monkeypatch those NumPy classes that you are using. If you need help with that (forruamel.yaml) post another question with the full program that dumps, what the output looks like, and what output you want
– Anthon
Dec 31 '18 at 18:25
add a comment |
No, your requirements cannot be satisfied.
You already have one function yaml.dump() that saves all those data types. As you noticed it doesn't do so in a very readable way for numpy data structures. This is caused by numpy not having dumping routines for their special data structure, instead falling back to the, not-so-readable, default !python.... tagged dump of the datastructure. Now you (or the YAML or Numpy package maintainers) can provide, special routines for those objects that dump in a more readable format, so that could be covered. You can make the representer in your YAML library more intelligent and get more readable output for Numpy datastructures without touching Numpy classes.
But you want this for all future datatypes, and IMO a variation of Gödel's theorem applies: even if the YAML library is extended so that it covers all known cases and dumps them in a readable way, there will always be new datastructures, especially in C based extensions (like Numpy), that cannot be represented in a readable way without extra work.
So because of your
Any other future unknown datatype that hasn't occurred to me yet.
premise, this is not just a lot of difficult work, but impossible.
answered Dec 31 '18 at 15:59
AnthonAnthon
29.9k1693147
Are you sure about the strict impossibility? In the end, any data structure would be comprised of other data structures, that, recursively, in the end would be some primitives. I assumed some library implemented that in some tree format, but why would it be impossible?
– Gulzar
Dec 31 '18 at 16:09
Also, what about without the "any future" requirement? Is there some library that handles the other use cases internally?
– Gulzar
Dec 31 '18 at 16:15
There is nothing in ruamel.yaml nor in PyYAML (and I don't think NumPy works with the syck library). In principle a third party library could provide better readable representations, but I don't think any library does. I once looked at doing that for ruamel.yaml, but I would have had to dig in too deeply and I am not a NumPy user, so the motivation for that is low.. You should be able monkeypatch those NumPy classes that you are using. If you need help with that (forruamel.yaml) post another question with the full program that dumps, what the output looks like, and what output you want
– Anthon
Dec 31 '18 at 18:25
add a comment |
No, your requirements cannot be satisfied.
You already have one function yaml.dump() that saves all those data types. As you noticed it doesn't do so in a very readable way for numpy data structures. This is caused by numpy not having dumping routines for their special data structure, instead falling back to the, not-so-readable, default !python.... tagged dump of the datastructure. Now you (or the YAML or Numpy package maintainers) can provide, special routines for those objects that dump in a more readable format, so that could be covered. You can make the representer in your YAML library more intelligent and get more readable output for Numpy datastructures without touching Numpy classes.
But you want this for all future datatypes, and IMO a variation of Gödel's theorem applies: even if the YAML library is extended so that it covers all known cases and dumps them in a readable way, there will always be new datastructures, especially in C based extensions (like Numpy), that cannot be represented in a readable way without extra work.
So because of your
Any other future unknown datatype that hasn't occurred to me yet.
premise, this is not just a lot of difficult work, but impossible.
answered Dec 31 '18 at 15:59
AnthonAnthon
29.9k1693147
No, your requirements cannot be satisfied.
You already have one function yaml.dump() that saves all those data types. As you noticed it doesn't do so in a very readable way for numpy data structures. This is caused by numpy not having dumping routines for their special data structure, instead falling back to the, not-so-readable, default !python.... tagged dump of the datastructure. Now you (or the YAML or Numpy package maintainers) can provide, special routines for those objects that dump in a more readable format, so that could be covered. You can make the representer in your YAML library more intelligent and get more readable output for Numpy datastructures without touching Numpy classes.
But you want this for all future datatypes, and IMO a variation of Gödel's theorem applies: even if the YAML library is extended so that it covers all known cases and dumps them in a readable way, there will always be new datastructures, especially in C based extensions (like Numpy), that cannot be represented in a readable way without extra work.
So because of your
Any other future unknown datatype that hasn't occurred to me yet.
premise, this is not just a lot of difficult work, but impossible.
answered Dec 31 '18 at 15:59
AnthonAnthon
29.9k1693147
answered Dec 31 '18 at 15:59
AnthonAnthon
29.9k1693147
answered Dec 31 '18 at 15:59
AnthonAnthon
29.9k1693147
answered Dec 31 '18 at 15:59
AnthonAnthon
29.9k1693147
29.9k1693147
Are you sure about the strict impossibility? In the end, any data structure would be comprised of other data structures, that, recursively, in the end would be some primitives. I assumed some library implemented that in some tree format, but why would it be impossible?
– Gulzar
Dec 31 '18 at 16:09
Also, what about without the "any future" requirement? Is there some library that handles the other use cases internally?
– Gulzar
Dec 31 '18 at 16:15
There is nothing in ruamel.yaml nor in PyYAML (and I don't think NumPy works with the syck library). In principle a third party library could provide better readable representations, but I don't think any library does. I once looked at doing that for ruamel.yaml, but I would have had to dig in too deeply and I am not a NumPy user, so the motivation for that is low.. You should be able monkeypatch those NumPy classes that you are using. If you need help with that (forruamel.yaml) post another question with the full program that dumps, what the output looks like, and what output you want
– Anthon
Dec 31 '18 at 18:25
add a comment |
Are you sure about the strict impossibility? In the end, any data structure would be comprised of other data structures, that, recursively, in the end would be some primitives. I assumed some library implemented that in some tree format, but why would it be impossible?
– Gulzar
Dec 31 '18 at 16:09
Also, what about without the "any future" requirement? Is there some library that handles the other use cases internally?
– Gulzar
Dec 31 '18 at 16:15
There is nothing in ruamel.yaml nor in PyYAML (and I don't think NumPy works with the syck library). In principle a third party library could provide better readable representations, but I don't think any library does. I once looked at doing that for ruamel.yaml, but I would have had to dig in too deeply and I am not a NumPy user, so the motivation for that is low.. You should be able monkeypatch those NumPy classes that you are using. If you need help with that (forruamel.yaml) post another question with the full program that dumps, what the output looks like, and what output you want
– Anthon
Dec 31 '18 at 18:25
Are you sure about the strict impossibility? In the end, any data structure would be comprised of other data structures, that, recursively, in the end would be some primitives. I assumed some library implemented that in some tree format, but why would it be impossible?
– Gulzar
Dec 31 '18 at 16:09
Are you sure about the strict impossibility? In the end, any data structure would be comprised of other data structures, that, recursively, in the end would be some primitives. I assumed some library implemented that in some tree format, but why would it be impossible?
– Gulzar
Dec 31 '18 at 16:09
Also, what about without the "any future" requirement? Is there some library that handles the other use cases internally?
– Gulzar
Dec 31 '18 at 16:15
Also, what about without the "any future" requirement? Is there some library that handles the other use cases internally?
– Gulzar
Dec 31 '18 at 16:15
There is nothing in ruamel.yaml nor in PyYAML (and I don't think NumPy works with the syck library). In principle a third party library could provide better readable representations, but I don't think any library does. I once looked at doing that for ruamel.yaml, but I would have had to dig in too deeply and I am not a NumPy user, so the motivation for that is low.. You should be able monkeypatch those NumPy classes that you are using. If you need help with that (for
ruamel.yaml) post another question with the full program that dumps, what the output looks like, and what output you want– Anthon
Dec 31 '18 at 18:25
There is nothing in ruamel.yaml nor in PyYAML (and I don't think NumPy works with the syck library). In principle a third party library could provide better readable representations, but I don't think any library does. I once looked at doing that for ruamel.yaml, but I would have had to dig in too deeply and I am not a NumPy user, so the motivation for that is low.. You should be able monkeypatch those NumPy classes that you are using. If you need help with that (for
ruamel.yaml) post another question with the full program that dumps, what the output looks like, and what output you want– Anthon
Dec 31 '18 at 18:25
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53987582%2fhow-to-write-to-yaml-file-in-human-readable-format%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
