echo without trimming the space in awk command

Multi tool use
Multi tool use












3















I have a file consisting of multiple rows like this



10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GRCHEMBUR      MHIN|0000000010000.00|6761857316|508998|6011|GL


I have to split and replace the column 11 into 4 different columns using the count of character.



This is the 11th column containing extra spaces also.



SHOP NO.5,6,7 RUNWAL GRCHEMBUR      MHIN


This is I have done



ls *.txt *.TXT| while read line
do
subName="$(cut -d'.' -f1 <<<"$line")"
awk -F"|" '{ "echo -n "$11" | cut -c1-23" | getline ton;
"echo -n "$11" | cut -c24-36" | getline city;
"echo -n "$11" | cut -c37-38" | getline state;
"echo -n "$11" | cut -c39-40" | getline country;
$11=ton"|"city"|"state"|"country; print $0

}' OFS="|" $line > $subName$output


done



But while doing echo of 11th column, its trimming the extra spaces which leads to mismatch in count of character. Is there any way to echo without trimming spaces ?



Actual output



10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR MHIN|||0000000010000.00|6761857316|508998|6011|GL


Expected Output



10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR|MH|IN|0000000010000.00|6761857316|508998|6011|GL









share|improve this question

























  • Comments are not for extended discussion; this conversation has been moved to chat.

    – Bhargav Rao
    Jan 3 at 9:08
















3















I have a file consisting of multiple rows like this



10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GRCHEMBUR      MHIN|0000000010000.00|6761857316|508998|6011|GL


I have to split and replace the column 11 into 4 different columns using the count of character.



This is the 11th column containing extra spaces also.



SHOP NO.5,6,7 RUNWAL GRCHEMBUR      MHIN


This is I have done



ls *.txt *.TXT| while read line
do
subName="$(cut -d'.' -f1 <<<"$line")"
awk -F"|" '{ "echo -n "$11" | cut -c1-23" | getline ton;
"echo -n "$11" | cut -c24-36" | getline city;
"echo -n "$11" | cut -c37-38" | getline state;
"echo -n "$11" | cut -c39-40" | getline country;
$11=ton"|"city"|"state"|"country; print $0

}' OFS="|" $line > $subName$output


done



But while doing echo of 11th column, its trimming the extra spaces which leads to mismatch in count of character. Is there any way to echo without trimming spaces ?



Actual output



10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR MHIN|||0000000010000.00|6761857316|508998|6011|GL


Expected Output



10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR|MH|IN|0000000010000.00|6761857316|508998|6011|GL









share|improve this question

























  • Comments are not for extended discussion; this conversation has been moved to chat.

    – Bhargav Rao
    Jan 3 at 9:08














3












3








3


1






I have a file consisting of multiple rows like this



10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GRCHEMBUR      MHIN|0000000010000.00|6761857316|508998|6011|GL


I have to split and replace the column 11 into 4 different columns using the count of character.



This is the 11th column containing extra spaces also.



SHOP NO.5,6,7 RUNWAL GRCHEMBUR      MHIN


This is I have done



ls *.txt *.TXT| while read line
do
subName="$(cut -d'.' -f1 <<<"$line")"
awk -F"|" '{ "echo -n "$11" | cut -c1-23" | getline ton;
"echo -n "$11" | cut -c24-36" | getline city;
"echo -n "$11" | cut -c37-38" | getline state;
"echo -n "$11" | cut -c39-40" | getline country;
$11=ton"|"city"|"state"|"country; print $0

}' OFS="|" $line > $subName$output


done



But while doing echo of 11th column, its trimming the extra spaces which leads to mismatch in count of character. Is there any way to echo without trimming spaces ?



Actual output



10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR MHIN|||0000000010000.00|6761857316|508998|6011|GL


Expected Output



10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR|MH|IN|0000000010000.00|6761857316|508998|6011|GL









share|improve this question
















I have a file consisting of multiple rows like this



10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GRCHEMBUR      MHIN|0000000010000.00|6761857316|508998|6011|GL


I have to split and replace the column 11 into 4 different columns using the count of character.



This is the 11th column containing extra spaces also.



SHOP NO.5,6,7 RUNWAL GRCHEMBUR      MHIN


This is I have done



ls *.txt *.TXT| while read line
do
subName="$(cut -d'.' -f1 <<<"$line")"
awk -F"|" '{ "echo -n "$11" | cut -c1-23" | getline ton;
"echo -n "$11" | cut -c24-36" | getline city;
"echo -n "$11" | cut -c37-38" | getline state;
"echo -n "$11" | cut -c39-40" | getline country;
$11=ton"|"city"|"state"|"country; print $0

}' OFS="|" $line > $subName$output


done



But while doing echo of 11th column, its trimming the extra spaces which leads to mismatch in count of character. Is there any way to echo without trimming spaces ?



Actual output



10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR MHIN|||0000000010000.00|6761857316|508998|6011|GL


Expected Output



10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR|MH|IN|0000000010000.00|6761857316|508998|6011|GL






linux bash






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Jan 2 at 16:26







Muddassir Rahman

















asked Jan 2 at 15:56









Muddassir RahmanMuddassir Rahman

235110




235110













  • Comments are not for extended discussion; this conversation has been moved to chat.

    – Bhargav Rao
    Jan 3 at 9:08



















  • Comments are not for extended discussion; this conversation has been moved to chat.

    – Bhargav Rao
    Jan 3 at 9:08

















Comments are not for extended discussion; this conversation has been moved to chat.

– Bhargav Rao
Jan 3 at 9:08





Comments are not for extended discussion; this conversation has been moved to chat.

– Bhargav Rao
Jan 3 at 9:08












2 Answers
2






active

oldest

votes


















2














A pure-bash implementation of all this logic



#!/usr/bin/env bash
shopt -s nocaseglob extglob
for f in *.txt; do
subName=${f%.*}
while IFS='|' read -r -a fields; do
location=${fields[10]}
ton=${location:0:23}; ton=${ton%%+([[:space:]])}
city=${location:23:12}; city=${city%%+([[:space:]])}
state=${location:36:2}
country=${location:38:2}
fields[10]="$ton|$city|$state|$country"
printf -v out '%s|' "${fields[@]}"
printf '%sn' "${out:0:$(( ${#out} - 1 ))}"
done <"$f" >"$subName.out"
done


It's slower (if I did this well, by about a factor of 10) than pure awk would be, but much faster than the awk/shell combination proposed in the question.





Going into the constructs used:




  • All the ${varname%...} and related constructs are parameter expansion. The specific ${varname%pattern} construct removes the shortest possible match for pattern from the value in varname, or the longest match if % is replaced with %%.

  • Using extglob enables extended globbing syntax, such as +([[:space:]]), which is equivalent to the regex syntax [[:space:]]+.






share|improve this answer

































    4














    The least annoying way to code this that I've found so far is:



    perl -F'|' -lane '$F[10] = join "|", unpack "a23 A13 a2 a2", $F[10]; print join "|", @F'


    It's fairly straightforward:




    • Iterate over lines of input; split each line on | and put the fields in @F.

    • For the 11th field ($F[10]), split it into fixed-width subfields using unpack (and trim trailing spaces from the second field (A instead of a)).

    • Reassemble subfields by joining with |.

    • Reassemble the whole line by joining with | and printing it.


    I haven't benchmarked it in any way, but it's likely much faster than the original code that spawns multiple shell and cut processes per input line because it's all done in one process.



    A complete solution would wrap it in a shell loop:



    for file in *.txt *.TXT; do
    outfile="${file%.*}$output"
    perl -F'|' -lane '...' "$file" > "$outfile"
    done


    Or if you don't need to trim the .txt part (and you don't have too many files to fit on the command line):



    perl -i.out -F'|' -lane '...' *.txt *.TXT


    This simply places the output for each input file foo.txt in foo.txt.out.






    share|improve this answer

























      Your Answer






      StackExchange.ifUsing("editor", function () {
      StackExchange.using("externalEditor", function () {
      StackExchange.using("snippets", function () {
      StackExchange.snippets.init();
      });
      });
      }, "code-snippets");

      StackExchange.ready(function() {
      var channelOptions = {
      tags: "".split(" "),
      id: "1"
      };
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function() {
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled) {
      StackExchange.using("snippets", function() {
      createEditor();
      });
      }
      else {
      createEditor();
      }
      });

      function createEditor() {
      StackExchange.prepareEditor({
      heartbeatType: 'answer',
      autoActivateHeartbeat: false,
      convertImagesToLinks: true,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: 10,
      bindNavPrevention: true,
      postfix: "",
      imageUploader: {
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      },
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      });


      }
      });














      draft saved

      draft discarded


















      StackExchange.ready(
      function () {
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54009349%2fecho-without-trimming-the-space-in-awk-command%23new-answer', 'question_page');
      }
      );

      Post as a guest















      Required, but never shown

























      2 Answers
      2






      active

      oldest

      votes








      2 Answers
      2






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      2














      A pure-bash implementation of all this logic



      #!/usr/bin/env bash
      shopt -s nocaseglob extglob
      for f in *.txt; do
      subName=${f%.*}
      while IFS='|' read -r -a fields; do
      location=${fields[10]}
      ton=${location:0:23}; ton=${ton%%+([[:space:]])}
      city=${location:23:12}; city=${city%%+([[:space:]])}
      state=${location:36:2}
      country=${location:38:2}
      fields[10]="$ton|$city|$state|$country"
      printf -v out '%s|' "${fields[@]}"
      printf '%sn' "${out:0:$(( ${#out} - 1 ))}"
      done <"$f" >"$subName.out"
      done


      It's slower (if I did this well, by about a factor of 10) than pure awk would be, but much faster than the awk/shell combination proposed in the question.





      Going into the constructs used:




      • All the ${varname%...} and related constructs are parameter expansion. The specific ${varname%pattern} construct removes the shortest possible match for pattern from the value in varname, or the longest match if % is replaced with %%.

      • Using extglob enables extended globbing syntax, such as +([[:space:]]), which is equivalent to the regex syntax [[:space:]]+.






      share|improve this answer






























        2














        A pure-bash implementation of all this logic



        #!/usr/bin/env bash
        shopt -s nocaseglob extglob
        for f in *.txt; do
        subName=${f%.*}
        while IFS='|' read -r -a fields; do
        location=${fields[10]}
        ton=${location:0:23}; ton=${ton%%+([[:space:]])}
        city=${location:23:12}; city=${city%%+([[:space:]])}
        state=${location:36:2}
        country=${location:38:2}
        fields[10]="$ton|$city|$state|$country"
        printf -v out '%s|' "${fields[@]}"
        printf '%sn' "${out:0:$(( ${#out} - 1 ))}"
        done <"$f" >"$subName.out"
        done


        It's slower (if I did this well, by about a factor of 10) than pure awk would be, but much faster than the awk/shell combination proposed in the question.





        Going into the constructs used:




        • All the ${varname%...} and related constructs are parameter expansion. The specific ${varname%pattern} construct removes the shortest possible match for pattern from the value in varname, or the longest match if % is replaced with %%.

        • Using extglob enables extended globbing syntax, such as +([[:space:]]), which is equivalent to the regex syntax [[:space:]]+.






        share|improve this answer




























          2












          2








          2







          A pure-bash implementation of all this logic



          #!/usr/bin/env bash
          shopt -s nocaseglob extglob
          for f in *.txt; do
          subName=${f%.*}
          while IFS='|' read -r -a fields; do
          location=${fields[10]}
          ton=${location:0:23}; ton=${ton%%+([[:space:]])}
          city=${location:23:12}; city=${city%%+([[:space:]])}
          state=${location:36:2}
          country=${location:38:2}
          fields[10]="$ton|$city|$state|$country"
          printf -v out '%s|' "${fields[@]}"
          printf '%sn' "${out:0:$(( ${#out} - 1 ))}"
          done <"$f" >"$subName.out"
          done


          It's slower (if I did this well, by about a factor of 10) than pure awk would be, but much faster than the awk/shell combination proposed in the question.





          Going into the constructs used:




          • All the ${varname%...} and related constructs are parameter expansion. The specific ${varname%pattern} construct removes the shortest possible match for pattern from the value in varname, or the longest match if % is replaced with %%.

          • Using extglob enables extended globbing syntax, such as +([[:space:]]), which is equivalent to the regex syntax [[:space:]]+.






          share|improve this answer















          A pure-bash implementation of all this logic



          #!/usr/bin/env bash
          shopt -s nocaseglob extglob
          for f in *.txt; do
          subName=${f%.*}
          while IFS='|' read -r -a fields; do
          location=${fields[10]}
          ton=${location:0:23}; ton=${ton%%+([[:space:]])}
          city=${location:23:12}; city=${city%%+([[:space:]])}
          state=${location:36:2}
          country=${location:38:2}
          fields[10]="$ton|$city|$state|$country"
          printf -v out '%s|' "${fields[@]}"
          printf '%sn' "${out:0:$(( ${#out} - 1 ))}"
          done <"$f" >"$subName.out"
          done


          It's slower (if I did this well, by about a factor of 10) than pure awk would be, but much faster than the awk/shell combination proposed in the question.





          Going into the constructs used:




          • All the ${varname%...} and related constructs are parameter expansion. The specific ${varname%pattern} construct removes the shortest possible match for pattern from the value in varname, or the longest match if % is replaced with %%.

          • Using extglob enables extended globbing syntax, such as +([[:space:]]), which is equivalent to the regex syntax [[:space:]]+.







          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited Jan 2 at 16:41

























          answered Jan 2 at 16:11









          Charles DuffyCharles Duffy

          179k25204259




          179k25204259

























              4














              The least annoying way to code this that I've found so far is:



              perl -F'|' -lane '$F[10] = join "|", unpack "a23 A13 a2 a2", $F[10]; print join "|", @F'


              It's fairly straightforward:




              • Iterate over lines of input; split each line on | and put the fields in @F.

              • For the 11th field ($F[10]), split it into fixed-width subfields using unpack (and trim trailing spaces from the second field (A instead of a)).

              • Reassemble subfields by joining with |.

              • Reassemble the whole line by joining with | and printing it.


              I haven't benchmarked it in any way, but it's likely much faster than the original code that spawns multiple shell and cut processes per input line because it's all done in one process.



              A complete solution would wrap it in a shell loop:



              for file in *.txt *.TXT; do
              outfile="${file%.*}$output"
              perl -F'|' -lane '...' "$file" > "$outfile"
              done


              Or if you don't need to trim the .txt part (and you don't have too many files to fit on the command line):



              perl -i.out -F'|' -lane '...' *.txt *.TXT


              This simply places the output for each input file foo.txt in foo.txt.out.






              share|improve this answer






























                4














                The least annoying way to code this that I've found so far is:



                perl -F'|' -lane '$F[10] = join "|", unpack "a23 A13 a2 a2", $F[10]; print join "|", @F'


                It's fairly straightforward:




                • Iterate over lines of input; split each line on | and put the fields in @F.

                • For the 11th field ($F[10]), split it into fixed-width subfields using unpack (and trim trailing spaces from the second field (A instead of a)).

                • Reassemble subfields by joining with |.

                • Reassemble the whole line by joining with | and printing it.


                I haven't benchmarked it in any way, but it's likely much faster than the original code that spawns multiple shell and cut processes per input line because it's all done in one process.



                A complete solution would wrap it in a shell loop:



                for file in *.txt *.TXT; do
                outfile="${file%.*}$output"
                perl -F'|' -lane '...' "$file" > "$outfile"
                done


                Or if you don't need to trim the .txt part (and you don't have too many files to fit on the command line):



                perl -i.out -F'|' -lane '...' *.txt *.TXT


                This simply places the output for each input file foo.txt in foo.txt.out.






                share|improve this answer




























                  4












                  4








                  4







                  The least annoying way to code this that I've found so far is:



                  perl -F'|' -lane '$F[10] = join "|", unpack "a23 A13 a2 a2", $F[10]; print join "|", @F'


                  It's fairly straightforward:




                  • Iterate over lines of input; split each line on | and put the fields in @F.

                  • For the 11th field ($F[10]), split it into fixed-width subfields using unpack (and trim trailing spaces from the second field (A instead of a)).

                  • Reassemble subfields by joining with |.

                  • Reassemble the whole line by joining with | and printing it.


                  I haven't benchmarked it in any way, but it's likely much faster than the original code that spawns multiple shell and cut processes per input line because it's all done in one process.



                  A complete solution would wrap it in a shell loop:



                  for file in *.txt *.TXT; do
                  outfile="${file%.*}$output"
                  perl -F'|' -lane '...' "$file" > "$outfile"
                  done


                  Or if you don't need to trim the .txt part (and you don't have too many files to fit on the command line):



                  perl -i.out -F'|' -lane '...' *.txt *.TXT


                  This simply places the output for each input file foo.txt in foo.txt.out.






                  share|improve this answer















                  The least annoying way to code this that I've found so far is:



                  perl -F'|' -lane '$F[10] = join "|", unpack "a23 A13 a2 a2", $F[10]; print join "|", @F'


                  It's fairly straightforward:




                  • Iterate over lines of input; split each line on | and put the fields in @F.

                  • For the 11th field ($F[10]), split it into fixed-width subfields using unpack (and trim trailing spaces from the second field (A instead of a)).

                  • Reassemble subfields by joining with |.

                  • Reassemble the whole line by joining with | and printing it.


                  I haven't benchmarked it in any way, but it's likely much faster than the original code that spawns multiple shell and cut processes per input line because it's all done in one process.



                  A complete solution would wrap it in a shell loop:



                  for file in *.txt *.TXT; do
                  outfile="${file%.*}$output"
                  perl -F'|' -lane '...' "$file" > "$outfile"
                  done


                  Or if you don't need to trim the .txt part (and you don't have too many files to fit on the command line):



                  perl -i.out -F'|' -lane '...' *.txt *.TXT


                  This simply places the output for each input file foo.txt in foo.txt.out.







                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited Jan 2 at 16:42

























                  answered Jan 2 at 16:32









                  melpomenemelpomene

                  61.7k54994




                  61.7k54994






























                      draft saved

                      draft discarded




















































                      Thanks for contributing an answer to Stack Overflow!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid



                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.


                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function () {
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54009349%2fecho-without-trimming-the-space-in-awk-command%23new-answer', 'question_page');
                      }
                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      3o4a rbKUmCy4 pqsn4z8 38uNSqwmq7xgJiuGVCq su5kg3QswKeAWJZ p,NuVxrHFh1FADC8EblaX,6Xk oYVD 8j,y,uPsfQLMV
                      X J2Qzp dJq6Z7G d,k,h3FrCMmTtFWbK3WVg,Kj D,AHkef Ln3vlS61q,qNqnKa,bNcQA1dxR4cD

                      Popular posts from this blog

                      Monofisismo

                      Angular Downloading a file using contenturl with Basic Authentication

                      Olmecas