Access File inside the dataflow pipeline
I want to download certain files from to the temp location before the pipeline starts.The files .mmdb files which are to be read in the ParDo fucntion.The files are stored on Google Storage but the method consuming the .mmdb files requires them to be a File(java.io) object.
If I include it in --filesToStage ,they are available as InputStream
inside the zip . I want to access them as files not InputStream.
What is the best way to achieve this?
I am currently downloading the files in a temporary folder on the worker inside the Setup of the ParDo .
google-cloud-dataflow apache-beam dataflow apache-beam-io
asked Jan 2 at 12:06
user3777228user3777228
598
add a comment |
I want to download certain files from to the temp location before the pipeline starts.The files .mmdb files which are to be read in the ParDo fucntion.The files are stored on Google Storage but the method consuming the .mmdb files requires them to be a File(java.io) object.
If I include it in --filesToStage ,they are available as InputStream
inside the zip . I want to access them as files not InputStream.
What is the best way to achieve this?
I am currently downloading the files in a temporary folder on the worker inside the Setup of the ParDo .
google-cloud-dataflow apache-beam dataflow apache-beam-io
asked Jan 2 at 12:06
user3777228user3777228
598
add a comment |
I want to download certain files from to the temp location before the pipeline starts.The files .mmdb files which are to be read in the ParDo fucntion.The files are stored on Google Storage but the method consuming the .mmdb files requires them to be a File(java.io) object.
If I include it in --filesToStage ,they are available as InputStream
inside the zip . I want to access them as files not InputStream.
What is the best way to achieve this?
I am currently downloading the files in a temporary folder on the worker inside the Setup of the ParDo .
google-cloud-dataflow apache-beam dataflow apache-beam-io
asked Jan 2 at 12:06
user3777228user3777228
598
I want to download certain files from to the temp location before the pipeline starts.The files .mmdb files which are to be read in the ParDo fucntion.The files are stored on Google Storage but the method consuming the .mmdb files requires them to be a File(java.io) object.
If I include it in --filesToStage ,they are available as InputStream
inside the zip . I want to access them as files not InputStream.
What is the best way to achieve this?
I am currently downloading the files in a temporary folder on the worker inside the Setup of the ParDo .
google-cloud-dataflow apache-beam dataflow apache-beam-io
google-cloud-dataflow apache-beam dataflow apache-beam-io
asked Jan 2 at 12:06
user3777228user3777228
598
asked Jan 2 at 12:06
user3777228user3777228
598
asked Jan 2 at 12:06
user3777228user3777228
598
asked Jan 2 at 12:06
user3777228user3777228
598
asked Jan 2 at 12:06
user3777228user3777228
598
598
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
This is a very broad and high level question. The answer depends on your logic that consumes the files. File represents a file on a filesystem so if you have a component that requires the input to be an instance of File then it is a correct thing to write it to a temp folder locally. Beam doesn't provide a better abstraction for this case.
However I would recommend to look into updating the logic that currently handles Files to accept other kinds of input as well. You likely hit the issue caused by the lack of separation of concerns and tight coupling. That is you have a component that takes in a File, opens it, deals with errors while opening it, reads it, parses data from it, maybe even validates and processes the data. All of these are separate concerns and probably should be handled by separate components that you can combine and replace together when needed, for example:
- a class that knows how to deal with a filesystem and turn a path into a byte stream;
- similar class that knows how to deal with getting a file over http (e.g. GCS use case) and turn it into a byte stream;
- a component that knows how to parse the byte stream into data;
- a component that processes the parsed data;
- other things can probably live anywhere;
This way you can easily implement any other sources for your component, compose and test them independently.
For example, you could implement your logic as 2 joined PCollections, one of which would read from the GCS location directly, parse the text lines, and and process it in the actual business logic before joining it with the other PCollection.
answered Jan 2 at 18:21
AntonAnton
1,177216
This File has to be read ideally once for each worker. The input to the pipeline is pub/sub.
– user3777228
Jan 4 at 12:59
I'm a little confused by your question, but have you seen this
– WIT
Jan 12 at 23:01
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54006072%2faccess-file-inside-the-dataflow-pipeline%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
This is a very broad and high level question. The answer depends on your logic that consumes the files. File represents a file on a filesystem so if you have a component that requires the input to be an instance of File then it is a correct thing to write it to a temp folder locally. Beam doesn't provide a better abstraction for this case.
However I would recommend to look into updating the logic that currently handles Files to accept other kinds of input as well. You likely hit the issue caused by the lack of separation of concerns and tight coupling. That is you have a component that takes in a File, opens it, deals with errors while opening it, reads it, parses data from it, maybe even validates and processes the data. All of these are separate concerns and probably should be handled by separate components that you can combine and replace together when needed, for example:
- a class that knows how to deal with a filesystem and turn a path into a byte stream;
- similar class that knows how to deal with getting a file over http (e.g. GCS use case) and turn it into a byte stream;
- a component that knows how to parse the byte stream into data;
- a component that processes the parsed data;
- other things can probably live anywhere;
This way you can easily implement any other sources for your component, compose and test them independently.
For example, you could implement your logic as 2 joined PCollections, one of which would read from the GCS location directly, parse the text lines, and and process it in the actual business logic before joining it with the other PCollection.
answered Jan 2 at 18:21
AntonAnton
1,177216
This File has to be read ideally once for each worker. The input to the pipeline is pub/sub.
– user3777228
Jan 4 at 12:59
I'm a little confused by your question, but have you seen this
– WIT
Jan 12 at 23:01
add a comment |
This is a very broad and high level question. The answer depends on your logic that consumes the files. File represents a file on a filesystem so if you have a component that requires the input to be an instance of File then it is a correct thing to write it to a temp folder locally. Beam doesn't provide a better abstraction for this case.
However I would recommend to look into updating the logic that currently handles Files to accept other kinds of input as well. You likely hit the issue caused by the lack of separation of concerns and tight coupling. That is you have a component that takes in a File, opens it, deals with errors while opening it, reads it, parses data from it, maybe even validates and processes the data. All of these are separate concerns and probably should be handled by separate components that you can combine and replace together when needed, for example:
- a class that knows how to deal with a filesystem and turn a path into a byte stream;
- similar class that knows how to deal with getting a file over http (e.g. GCS use case) and turn it into a byte stream;
- a component that knows how to parse the byte stream into data;
- a component that processes the parsed data;
- other things can probably live anywhere;
This way you can easily implement any other sources for your component, compose and test them independently.
For example, you could implement your logic as 2 joined PCollections, one of which would read from the GCS location directly, parse the text lines, and and process it in the actual business logic before joining it with the other PCollection.
answered Jan 2 at 18:21
AntonAnton
1,177216
This File has to be read ideally once for each worker. The input to the pipeline is pub/sub.
– user3777228
Jan 4 at 12:59
I'm a little confused by your question, but have you seen this
– WIT
Jan 12 at 23:01
add a comment |
This is a very broad and high level question. The answer depends on your logic that consumes the files. File represents a file on a filesystem so if you have a component that requires the input to be an instance of File then it is a correct thing to write it to a temp folder locally. Beam doesn't provide a better abstraction for this case.
However I would recommend to look into updating the logic that currently handles Files to accept other kinds of input as well. You likely hit the issue caused by the lack of separation of concerns and tight coupling. That is you have a component that takes in a File, opens it, deals with errors while opening it, reads it, parses data from it, maybe even validates and processes the data. All of these are separate concerns and probably should be handled by separate components that you can combine and replace together when needed, for example:
- a class that knows how to deal with a filesystem and turn a path into a byte stream;
- similar class that knows how to deal with getting a file over http (e.g. GCS use case) and turn it into a byte stream;
- a component that knows how to parse the byte stream into data;
- a component that processes the parsed data;
- other things can probably live anywhere;
This way you can easily implement any other sources for your component, compose and test them independently.
For example, you could implement your logic as 2 joined PCollections, one of which would read from the GCS location directly, parse the text lines, and and process it in the actual business logic before joining it with the other PCollection.
answered Jan 2 at 18:21
AntonAnton
1,177216
This is a very broad and high level question. The answer depends on your logic that consumes the files. File represents a file on a filesystem so if you have a component that requires the input to be an instance of File then it is a correct thing to write it to a temp folder locally. Beam doesn't provide a better abstraction for this case.
However I would recommend to look into updating the logic that currently handles Files to accept other kinds of input as well. You likely hit the issue caused by the lack of separation of concerns and tight coupling. That is you have a component that takes in a File, opens it, deals with errors while opening it, reads it, parses data from it, maybe even validates and processes the data. All of these are separate concerns and probably should be handled by separate components that you can combine and replace together when needed, for example:
- a class that knows how to deal with a filesystem and turn a path into a byte stream;
- similar class that knows how to deal with getting a file over http (e.g. GCS use case) and turn it into a byte stream;
- a component that knows how to parse the byte stream into data;
- a component that processes the parsed data;
- other things can probably live anywhere;
This way you can easily implement any other sources for your component, compose and test them independently.
For example, you could implement your logic as 2 joined PCollections, one of which would read from the GCS location directly, parse the text lines, and and process it in the actual business logic before joining it with the other PCollection.
answered Jan 2 at 18:21
AntonAnton
1,177216
answered Jan 2 at 18:21
AntonAnton
1,177216
answered Jan 2 at 18:21
AntonAnton
1,177216
answered Jan 2 at 18:21
AntonAnton
1,177216
1,177216
This File has to be read ideally once for each worker. The input to the pipeline is pub/sub.
– user3777228
Jan 4 at 12:59
I'm a little confused by your question, but have you seen this
– WIT
Jan 12 at 23:01
add a comment |
This File has to be read ideally once for each worker. The input to the pipeline is pub/sub.
– user3777228
Jan 4 at 12:59
I'm a little confused by your question, but have you seen this
– WIT
Jan 12 at 23:01
This File has to be read ideally once for each worker. The input to the pipeline is pub/sub.
– user3777228
Jan 4 at 12:59
This File has to be read ideally once for each worker. The input to the pipeline is pub/sub.
– user3777228
Jan 4 at 12:59
I'm a little confused by your question, but have you seen this
– WIT
Jan 12 at 23:01
I'm a little confused by your question, but have you seen this
– WIT
Jan 12 at 23:01
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54006072%2faccess-file-inside-the-dataflow-pipeline%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
