how to use read and write stream of csv-parse





.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}







0















hello guys i m new over here and got confuse how i can use readstream and write stream right now i tried this(i m using this library https://www.npmjs.com/package/csv-parse) 



    fs.createReadStream(path.join(__dirname,"../demofile/Customers.csv"))

        .pipe(parse(options))

        .on('data', function(csvrow) {

            console.log(csvrow);

            //do something with csvrow

            output.push(csvrow);        

        })

        .on('end',function() {

        //do something wiht csvData

        console.log(csvData);

        });


but i don't think it correct way read and write stream of csv-parse can anyone suggest me how i can use read and write with above code. basically i want to read a csv file and write it into another csv file reason for using read and write stream is because my file size is about 2gb






node.js express node-csv-parse







share|improve this question























  • Can you show the rest of your code, I'm not sure what "output" and "csvData" are?

    – Elliot Blackburn
    Jan 4 at 12:03











  • You will need to show what your two CSV files look like and what the desired output of the combined CSV file should be.

    – Francisco Mateo
    Jan 4 at 12:19




















0















hello guys i m new over here and got confuse how i can use readstream and write stream right now i tried this(i m using this library https://www.npmjs.com/package/csv-parse) 



    fs.createReadStream(path.join(__dirname,"../demofile/Customers.csv"))

        .pipe(parse(options))

        .on('data', function(csvrow) {

            console.log(csvrow);

            //do something with csvrow

            output.push(csvrow);        

        })

        .on('end',function() {

        //do something wiht csvData

        console.log(csvData);

        });


but i don't think it correct way read and write stream of csv-parse can anyone suggest me how i can use read and write with above code. basically i want to read a csv file and write it into another csv file reason for using read and write stream is because my file size is about 2gb






node.js express node-csv-parse







share|improve this question























  • Can you show the rest of your code, I'm not sure what "output" and "csvData" are?

    – Elliot Blackburn
    Jan 4 at 12:03











  • You will need to show what your two CSV files look like and what the desired output of the combined CSV file should be.

    – Francisco Mateo
    Jan 4 at 12:19
















0












0








0








hello guys i m new over here and got confuse how i can use readstream and write stream right now i tried this(i m using this library https://www.npmjs.com/package/csv-parse) 



    fs.createReadStream(path.join(__dirname,"../demofile/Customers.csv"))

        .pipe(parse(options))

        .on('data', function(csvrow) {

            console.log(csvrow);

            //do something with csvrow

            output.push(csvrow);        

        })

        .on('end',function() {

        //do something wiht csvData

        console.log(csvData);

        });


but i don't think it correct way read and write stream of csv-parse can anyone suggest me how i can use read and write with above code. basically i want to read a csv file and write it into another csv file reason for using read and write stream is because my file size is about 2gb






node.js express node-csv-parse







share|improve this question














hello guys i m new over here and got confuse how i can use readstream and write stream right now i tried this(i m using this library https://www.npmjs.com/package/csv-parse) 



    fs.createReadStream(path.join(__dirname,"../demofile/Customers.csv"))

        .pipe(parse(options))

        .on('data', function(csvrow) {

            console.log(csvrow);

            //do something with csvrow

            output.push(csvrow);        

        })

        .on('end',function() {

        //do something wiht csvData

        console.log(csvData);

        });


but i don't think it correct way read and write stream of csv-parse can anyone suggest me how i can use read and write with above code. basically i want to read a csv file and write it into another csv file reason for using read and write stream is because my file size is about 2gb






node.js express node-csv-parse




node.js express node-csv-parse






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Jan 4 at 7:46









RockyRocky

155214




155214













  • Can you show the rest of your code, I'm not sure what "output" and "csvData" are?

    – Elliot Blackburn
    Jan 4 at 12:03











  • You will need to show what your two CSV files look like and what the desired output of the combined CSV file should be.

    – Francisco Mateo
    Jan 4 at 12:19





















  • Can you show the rest of your code, I'm not sure what "output" and "csvData" are?

    – Elliot Blackburn
    Jan 4 at 12:03











  • You will need to show what your two CSV files look like and what the desired output of the combined CSV file should be.

    – Francisco Mateo
    Jan 4 at 12:19



















Can you show the rest of your code, I'm not sure what "output" and "csvData" are?

– Elliot Blackburn
Jan 4 at 12:03





Can you show the rest of your code, I'm not sure what "output" and "csvData" are?

– Elliot Blackburn
Jan 4 at 12:03













You will need to show what your two CSV files look like and what the desired output of the combined CSV file should be.

– Francisco Mateo
Jan 4 at 12:19







You will need to show what your two CSV files look like and what the desired output of the combined CSV file should be.

– Francisco Mateo
Jan 4 at 12:19














1 Answer
1






active

oldest

votes


















1














The following code uses streams to add a new column. The file I have used was about 500MB and the maximum utilized RAM was less than 50MB. Note that instead of the csv-parse I have imported csv which is actually an 'umbrella' module that includes csv-parse, csv-generate, stream-transform and csv-stringify.



const fs = require('fs');

const csv = require('csv');

const path = require('path');

const EOL = require('os').EOL;



FILE = path.join(__dirname, 'IN.csv');

NEW_FILE = path.join(__dirname, 'OUT.csv');



const readStream = fs.createReadStream(FILE);

const writeStream = fs.createWriteStream(NEW_FILE);



const parse = csv.parse();



const transform = csv.transform((row, cb) => {



    row.push('NEW_COL');

    result = row.join(',') + EOL;

    cb(null, result);

});



readStream.pipe(parse).pipe(transform).pipe(writeStream);





share|improve this answer
























    Your Answer






    StackExchange.ifUsing("editor", function () {
    StackExchange.using("externalEditor", function () {
    StackExchange.using("snippets", function () {
    StackExchange.snippets.init();
    });
    });
    }, "code-snippets");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "1"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54034911%2fhow-to-use-read-and-write-stream-of-csv-parse%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    1














    The following code uses streams to add a new column. The file I have used was about 500MB and the maximum utilized RAM was less than 50MB. Note that instead of the csv-parse I have imported csv which is actually an 'umbrella' module that includes csv-parse, csv-generate, stream-transform and csv-stringify.



    const fs = require('fs');
    
const csv = require('csv');
    
const path = require('path');
    
const EOL = require('os').EOL;
    

    
FILE = path.join(__dirname, 'IN.csv');
    
NEW_FILE = path.join(__dirname, 'OUT.csv');
    

    
const readStream = fs.createReadStream(FILE);
    
const writeStream = fs.createWriteStream(NEW_FILE);
    

    
const parse = csv.parse();
    

    
const transform = csv.transform((row, cb) => {
    

    
    row.push('NEW_COL');
    
    result = row.join(',') + EOL;
    
    cb(null, result);
    
});
    

    
readStream.pipe(parse).pipe(transform).pipe(writeStream);





    share|improve this answer




























      1














      The following code uses streams to add a new column. The file I have used was about 500MB and the maximum utilized RAM was less than 50MB. Note that instead of the csv-parse I have imported csv which is actually an 'umbrella' module that includes csv-parse, csv-generate, stream-transform and csv-stringify.



      const fs = require('fs');
      
const csv = require('csv');
      
const path = require('path');
      
const EOL = require('os').EOL;
      

      
FILE = path.join(__dirname, 'IN.csv');
      
NEW_FILE = path.join(__dirname, 'OUT.csv');
      

      
const readStream = fs.createReadStream(FILE);
      
const writeStream = fs.createWriteStream(NEW_FILE);
      

      
const parse = csv.parse();
      

      
const transform = csv.transform((row, cb) => {
      

      
    row.push('NEW_COL');
      
    result = row.join(',') + EOL;
      
    cb(null, result);
      
});
      

      
readStream.pipe(parse).pipe(transform).pipe(writeStream);





      share|improve this answer


























        1












        1








        1







        The following code uses streams to add a new column. The file I have used was about 500MB and the maximum utilized RAM was less than 50MB. Note that instead of the csv-parse I have imported csv which is actually an 'umbrella' module that includes csv-parse, csv-generate, stream-transform and csv-stringify.



        const fs = require('fs');
        
const csv = require('csv');
        
const path = require('path');
        
const EOL = require('os').EOL;
        

        
FILE = path.join(__dirname, 'IN.csv');
        
NEW_FILE = path.join(__dirname, 'OUT.csv');
        

        
const readStream = fs.createReadStream(FILE);
        
const writeStream = fs.createWriteStream(NEW_FILE);
        

        
const parse = csv.parse();
        

        
const transform = csv.transform((row, cb) => {
        

        
    row.push('NEW_COL');
        
    result = row.join(',') + EOL;
        
    cb(null, result);
        
});
        

        
readStream.pipe(parse).pipe(transform).pipe(writeStream);





        share|improve this answer













        The following code uses streams to add a new column. The file I have used was about 500MB and the maximum utilized RAM was less than 50MB. Note that instead of the csv-parse I have imported csv which is actually an 'umbrella' module that includes csv-parse, csv-generate, stream-transform and csv-stringify.



        const fs = require('fs');
        
const csv = require('csv');
        
const path = require('path');
        
const EOL = require('os').EOL;
        

        
FILE = path.join(__dirname, 'IN.csv');
        
NEW_FILE = path.join(__dirname, 'OUT.csv');
        

        
const readStream = fs.createReadStream(FILE);
        
const writeStream = fs.createWriteStream(NEW_FILE);
        

        
const parse = csv.parse();
        

        
const transform = csv.transform((row, cb) => {
        

        
    row.push('NEW_COL');
        
    result = row.join(',') + EOL;
        
    cb(null, result);
        
});
        

        
readStream.pipe(parse).pipe(transform).pipe(writeStream);






        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Jan 4 at 12:32









        MrfksIVMrfksIV

        404211




        404211
































            draft saved

            draft discarded




















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid …



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54034911%2fhow-to-use-read-and-write-stream-of-csv-parse%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            -

            Popular posts from this blog

            Monofisismo

            Image
            Monofisismo (do grego: monos - "único, singular" e physis - "natureza") é o ponto de vista cristológico que defende que, depois da união do divino e do humano na encarnação histórica, Jesus Cristo, como encarnação do Filho ou Verbo ( Logos ) de Deus, teria apenas uma única "natureza", a divina, e não uma síntese de ambas. O monofisismo é contraposto pelo diofisismo (ou "diafisismo"), que defende que Jesus preservou em si as duas naturezas. Historicamente, o monofisismo se refere primordialmente à posição dos que (especialmente no Egito e, em menor grau, na Síria) rejeitaram as decisões do Concílio de Calcedônia em 451 (o quarto concílio ecumênico). Os membros mais moderados entre eles, porém, defendem a teologia "miafisita", que se tornou a oficial para as Igrejas Ortodoxas Orientais. Muitos ortodoxos orientais, porém, rejeitam essa classificação, mesmo como um termo genérico, mas ele é amplamente utilizado na literatura históri...
            Read more

            Angular Downloading a file using contenturl with Basic Authentication

            Image
            0 I have a contentUrl when user clicks on that,file should be downloaded. If I try to browse the contentUrl in the browser it asks for Userid and password.I believe this is Basic Authentication.I dont want user to enter UserId and Password so I am passing it Authorization(Please see below code). Url looks some thing like this: http://XXX.XXX.XX.XXX:8080/flowable-rest/service/runtime/tasks/90875/attachments/90555/content let username : string = 'admin'; let pwd : string = 'test'; let headers = new Headers(); headers.set('Accept', 'text/json'); headers.append('Content-Type', 'application/json'); headers.set('Access-Control-Allow-Origin', '*'); headers.set('Access-Control-Allow-Methods','POST, GET, OPTIONS, PUT, DELETE'); head...
            Read more

            Olmecas

            Image
            Monumento 1, uma das quatro cabeças colossais encontradas em La Venta, com altura aproximada de 3 metros Olmecas é a designação do povo e da civilização que estiveram na origem da antiga cultura pré-colombiana da Mesoamérica e que se desenvolveram nas regiões tropicais do centro-sul do atual México durante o pré-clássico, próximo de onde hoje estão localizados os estados mexicanos de Veracruz e Tabasco, no Istmo de Tehuantepec, numa zona designada área nuclear olmeca. A cultura olmeca floresceu nesta região aproximadamente entre 1500 e 400 a.C., [ 1 ] e crê-se que tenha sido a civilização-mãe de todas as civilizações mesoamericanas que se desenvolveram posteriormente. [ 2 ] No entanto, desconhece-se a sua exacta filiação étnica, ainda que existam numerosas hipóteses colocadas para tentar resolver esta questão. O etnónimo olmeca foi cunhado pelos arqueólogos do século XX, e não devem confundir-se com os muito posteriores olmecas-xicalancas que ocuparam vários locais do México...
            Read more