Converting Data Stored in AWS back into original format in R
I've been trying to use the AWS S3 storage option via R. I've been using the aws.s3 package to help do that.
Everything seems to work on until I try to recall and use an rds file I had saved on AWS.
By way of example:
library("aws.s3")
Sys.setenv("AWS_ACCESS_KEY_ID" = "mykey",
"AWS_SECRET_ACCESS_KEY" = "mysecretkey",
"AWS_DEFAULT_REGION" = "us-east-1",
"AWS_SESSION_TOKEN" = "mytoken")
#Create Dummy Data
testdata <- rep(1:3, 10)
#Save to AWS
s3saveRDS(testdata, object = "testdata.rds", bucket = "mybucket")
#Recall from AWS
newtestdata <- get_object("testdata.rds", bucket = "mybucket")
newtestdata comes back in a raw format but I can't find how to convert it into its original format. I've tried things such as rawToChar() but I get errors.
For info this is what the newtestdata file looks like in its raw form:
1f 8b 08 00 00 00 00 00 00 06 8b e0 62 60 60 60 62 60 66 61 64 60 62 06 32 19 78 81 58 0e 88 19 c1 e2 0c 0c cc f4 64 03 00 62 4b 7d f5 8e 00 00 00
What should I do to convert this file back to its original form?
r amazon-web-services
asked Dec 27 at 13:54
Will T-E
146110
|
show 2 more comments
I've been trying to use the AWS S3 storage option via R. I've been using the aws.s3 package to help do that.
Everything seems to work on until I try to recall and use an rds file I had saved on AWS.
By way of example:
library("aws.s3")
Sys.setenv("AWS_ACCESS_KEY_ID" = "mykey",
"AWS_SECRET_ACCESS_KEY" = "mysecretkey",
"AWS_DEFAULT_REGION" = "us-east-1",
"AWS_SESSION_TOKEN" = "mytoken")
#Create Dummy Data
testdata <- rep(1:3, 10)
#Save to AWS
s3saveRDS(testdata, object = "testdata.rds", bucket = "mybucket")
#Recall from AWS
newtestdata <- get_object("testdata.rds", bucket = "mybucket")
newtestdata comes back in a raw format but I can't find how to convert it into its original format. I've tried things such as rawToChar() but I get errors.
For info this is what the newtestdata file looks like in its raw form:
1f 8b 08 00 00 00 00 00 00 06 8b e0 62 60 60 60 62 60 66 61 64 60 62 06 32 19 78 81 58 0e 88 19 c1 e2 0c 0c cc f4 64 03 00 62 4b 7d f5 8e 00 00 00
What should I do to convert this file back to its original form?
r amazon-web-services
asked Dec 27 at 13:54
Will T-E
146110
When you runclass(newtestdata), what is the output? Is there perhaps another wrapper function which you need to call after retrieving a raw object from an S3 bucket?
– Tim Biegeleisen
Dec 27 at 14:05
@TimBiegeleisen it just says"raw"
– Will T-E
Dec 27 at 14:07
There are also noattributesto the object. If I tryrawConnection(newtestdata)I get: A connection with description "newtestdata" class "rawConnection" mode "r" text "binary" opened "opened" can read "yes" can write "no"
– Will T-E
Dec 27 at 14:09
1
Please read the official documentation for this package. I can't fix your problem with having R and S3 in front of me, which I unfortunately don't have right now.
– Tim Biegeleisen
Dec 27 at 14:11
1
Can you try s3readRDS(object = "mtcars.rds", bucket = "myexamplebucket") as mentioned in rdocumentation.org/packages/aws.s3/versions/0.3.12/topics/… and see if it they are matching with identical(mtcars, mtcars2)
– bdcloud
Dec 27 at 15:45
|
show 2 more comments
I've been trying to use the AWS S3 storage option via R. I've been using the aws.s3 package to help do that.
Everything seems to work on until I try to recall and use an rds file I had saved on AWS.
By way of example:
library("aws.s3")
Sys.setenv("AWS_ACCESS_KEY_ID" = "mykey",
"AWS_SECRET_ACCESS_KEY" = "mysecretkey",
"AWS_DEFAULT_REGION" = "us-east-1",
"AWS_SESSION_TOKEN" = "mytoken")
#Create Dummy Data
testdata <- rep(1:3, 10)
#Save to AWS
s3saveRDS(testdata, object = "testdata.rds", bucket = "mybucket")
#Recall from AWS
newtestdata <- get_object("testdata.rds", bucket = "mybucket")
newtestdata comes back in a raw format but I can't find how to convert it into its original format. I've tried things such as rawToChar() but I get errors.
For info this is what the newtestdata file looks like in its raw form:
1f 8b 08 00 00 00 00 00 00 06 8b e0 62 60 60 60 62 60 66 61 64 60 62 06 32 19 78 81 58 0e 88 19 c1 e2 0c 0c cc f4 64 03 00 62 4b 7d f5 8e 00 00 00
What should I do to convert this file back to its original form?
r amazon-web-services
asked Dec 27 at 13:54
Will T-E
146110
I've been trying to use the AWS S3 storage option via R. I've been using the aws.s3 package to help do that.
Everything seems to work on until I try to recall and use an rds file I had saved on AWS.
By way of example:
library("aws.s3")
Sys.setenv("AWS_ACCESS_KEY_ID" = "mykey",
"AWS_SECRET_ACCESS_KEY" = "mysecretkey",
"AWS_DEFAULT_REGION" = "us-east-1",
"AWS_SESSION_TOKEN" = "mytoken")
#Create Dummy Data
testdata <- rep(1:3, 10)
#Save to AWS
s3saveRDS(testdata, object = "testdata.rds", bucket = "mybucket")
#Recall from AWS
newtestdata <- get_object("testdata.rds", bucket = "mybucket")
newtestdata comes back in a raw format but I can't find how to convert it into its original format. I've tried things such as rawToChar() but I get errors.
For info this is what the newtestdata file looks like in its raw form:
1f 8b 08 00 00 00 00 00 00 06 8b e0 62 60 60 60 62 60 66 61 64 60 62 06 32 19 78 81 58 0e 88 19 c1 e2 0c 0c cc f4 64 03 00 62 4b 7d f5 8e 00 00 00
What should I do to convert this file back to its original form?
r amazon-web-services
r amazon-web-services
asked Dec 27 at 13:54
Will T-E
146110
asked Dec 27 at 13:54
Will T-E
146110
asked Dec 27 at 13:54
Will T-E
146110
asked Dec 27 at 13:54
Will T-E
146110
asked Dec 27 at 13:54
Will T-E
146110
146110
When you runclass(newtestdata), what is the output? Is there perhaps another wrapper function which you need to call after retrieving a raw object from an S3 bucket?
– Tim Biegeleisen
Dec 27 at 14:05
@TimBiegeleisen it just says"raw"
– Will T-E
Dec 27 at 14:07
There are also noattributesto the object. If I tryrawConnection(newtestdata)I get: A connection with description "newtestdata" class "rawConnection" mode "r" text "binary" opened "opened" can read "yes" can write "no"
– Will T-E
Dec 27 at 14:09
1
Please read the official documentation for this package. I can't fix your problem with having R and S3 in front of me, which I unfortunately don't have right now.
– Tim Biegeleisen
Dec 27 at 14:11
1
Can you try s3readRDS(object = "mtcars.rds", bucket = "myexamplebucket") as mentioned in rdocumentation.org/packages/aws.s3/versions/0.3.12/topics/… and see if it they are matching with identical(mtcars, mtcars2)
– bdcloud
Dec 27 at 15:45
|
show 2 more comments
When you runclass(newtestdata), what is the output? Is there perhaps another wrapper function which you need to call after retrieving a raw object from an S3 bucket?
– Tim Biegeleisen
Dec 27 at 14:05
@TimBiegeleisen it just says"raw"
– Will T-E
Dec 27 at 14:07
There are also noattributesto the object. If I tryrawConnection(newtestdata)I get: A connection with description "newtestdata" class "rawConnection" mode "r" text "binary" opened "opened" can read "yes" can write "no"
– Will T-E
Dec 27 at 14:09
1
Please read the official documentation for this package. I can't fix your problem with having R and S3 in front of me, which I unfortunately don't have right now.
– Tim Biegeleisen
Dec 27 at 14:11
1
Can you try s3readRDS(object = "mtcars.rds", bucket = "myexamplebucket") as mentioned in rdocumentation.org/packages/aws.s3/versions/0.3.12/topics/… and see if it they are matching with identical(mtcars, mtcars2)
– bdcloud
Dec 27 at 15:45
When you run
class(newtestdata), what is the output? Is there perhaps another wrapper function which you need to call after retrieving a raw object from an S3 bucket?– Tim Biegeleisen
Dec 27 at 14:05
When you run
class(newtestdata), what is the output? Is there perhaps another wrapper function which you need to call after retrieving a raw object from an S3 bucket?– Tim Biegeleisen
Dec 27 at 14:05
@TimBiegeleisen it just says
"raw"– Will T-E
Dec 27 at 14:07
@TimBiegeleisen it just says
"raw"– Will T-E
Dec 27 at 14:07
There are also no
attributes to the object. If I try rawConnection(newtestdata) I get: A connection with description "newtestdata" class "rawConnection" mode "r" text "binary" opened "opened" can read "yes" can write "no"– Will T-E
Dec 27 at 14:09
There are also no
attributes to the object. If I try rawConnection(newtestdata) I get: A connection with description "newtestdata" class "rawConnection" mode "r" text "binary" opened "opened" can read "yes" can write "no"– Will T-E
Dec 27 at 14:09
1
1
Please read the official documentation for this package. I can't fix your problem with having R and S3 in front of me, which I unfortunately don't have right now.
– Tim Biegeleisen
Dec 27 at 14:11
Please read the official documentation for this package. I can't fix your problem with having R and S3 in front of me, which I unfortunately don't have right now.
– Tim Biegeleisen
Dec 27 at 14:11
1
1
Can you try s3readRDS(object = "mtcars.rds", bucket = "myexamplebucket") as mentioned in rdocumentation.org/packages/aws.s3/versions/0.3.12/topics/… and see if it they are matching with identical(mtcars, mtcars2)
– bdcloud
Dec 27 at 15:45
Can you try s3readRDS(object = "mtcars.rds", bucket = "myexamplebucket") as mentioned in rdocumentation.org/packages/aws.s3/versions/0.3.12/topics/… and see if it they are matching with identical(mtcars, mtcars2)
– bdcloud
Dec 27 at 15:45
|
show 2 more comments
1 Answer
1
active
oldest
votes
You can try below snippet to read the data as it is as mentioned in [1] and see if it they are matching with identical() .
s3readRDS(object = "mtcars.rds", bucket = "myexamplebucket")
identical(mtcars, mtcars2)
answered Dec 27 at 15:59
bdcloud
401310
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53946225%2fconverting-data-stored-in-aws-back-into-original-format-in-r%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can try below snippet to read the data as it is as mentioned in [1] and see if it they are matching with identical() .
s3readRDS(object = "mtcars.rds", bucket = "myexamplebucket")
identical(mtcars, mtcars2)
answered Dec 27 at 15:59
bdcloud
401310
add a comment |
You can try below snippet to read the data as it is as mentioned in [1] and see if it they are matching with identical() .
s3readRDS(object = "mtcars.rds", bucket = "myexamplebucket")
identical(mtcars, mtcars2)
answered Dec 27 at 15:59
bdcloud
401310
add a comment |
You can try below snippet to read the data as it is as mentioned in [1] and see if it they are matching with identical() .
s3readRDS(object = "mtcars.rds", bucket = "myexamplebucket")
identical(mtcars, mtcars2)
answered Dec 27 at 15:59
bdcloud
401310
You can try below snippet to read the data as it is as mentioned in [1] and see if it they are matching with identical() .
s3readRDS(object = "mtcars.rds", bucket = "myexamplebucket")
identical(mtcars, mtcars2)
answered Dec 27 at 15:59
bdcloud
401310
answered Dec 27 at 15:59
bdcloud
401310
answered Dec 27 at 15:59
bdcloud
401310
answered Dec 27 at 15:59
bdcloud
401310
401310
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53946225%2fconverting-data-stored-in-aws-back-into-original-format-in-r%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
When you run
class(newtestdata), what is the output? Is there perhaps another wrapper function which you need to call after retrieving a raw object from an S3 bucket?– Tim Biegeleisen
Dec 27 at 14:05
@TimBiegeleisen it just says
"raw"– Will T-E
Dec 27 at 14:07
There are also no
attributesto the object. If I tryrawConnection(newtestdata)I get: A connection with description "newtestdata" class "rawConnection" mode "r" text "binary" opened "opened" can read "yes" can write "no"– Will T-E
Dec 27 at 14:09
1
Please read the official documentation for this package. I can't fix your problem with having R and S3 in front of me, which I unfortunately don't have right now.
– Tim Biegeleisen
Dec 27 at 14:11
1
Can you try s3readRDS(object = "mtcars.rds", bucket = "myexamplebucket") as mentioned in rdocumentation.org/packages/aws.s3/versions/0.3.12/topics/… and see if it they are matching with identical(mtcars, mtcars2)
– bdcloud
Dec 27 at 15:45