echo without trimming the space in awk command

Multi tool use
I have a file consisting of multiple rows like this
10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GRCHEMBUR MHIN|0000000010000.00|6761857316|508998|6011|GL
I have to split and replace the column 11 into 4 different columns using the count of character.
This is the 11th column containing extra spaces also.
SHOP NO.5,6,7 RUNWAL GRCHEMBUR MHIN
This is I have done
ls *.txt *.TXT| while read line
do
subName="$(cut -d'.' -f1 <<<"$line")"
awk -F"|" '{ "echo -n "$11" | cut -c1-23" | getline ton;
"echo -n "$11" | cut -c24-36" | getline city;
"echo -n "$11" | cut -c37-38" | getline state;
"echo -n "$11" | cut -c39-40" | getline country;
$11=ton"|"city"|"state"|"country; print $0
}' OFS="|" $line > $subName$output
done
But while doing echo of 11th column, its trimming the extra spaces which leads to mismatch in count of character. Is there any way to echo without trimming spaces ?
Actual output
10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR MHIN|||0000000010000.00|6761857316|508998|6011|GL
Expected Output
10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR|MH|IN|0000000010000.00|6761857316|508998|6011|GL
linux bash
add a comment |
I have a file consisting of multiple rows like this
10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GRCHEMBUR MHIN|0000000010000.00|6761857316|508998|6011|GL
I have to split and replace the column 11 into 4 different columns using the count of character.
This is the 11th column containing extra spaces also.
SHOP NO.5,6,7 RUNWAL GRCHEMBUR MHIN
This is I have done
ls *.txt *.TXT| while read line
do
subName="$(cut -d'.' -f1 <<<"$line")"
awk -F"|" '{ "echo -n "$11" | cut -c1-23" | getline ton;
"echo -n "$11" | cut -c24-36" | getline city;
"echo -n "$11" | cut -c37-38" | getline state;
"echo -n "$11" | cut -c39-40" | getline country;
$11=ton"|"city"|"state"|"country; print $0
}' OFS="|" $line > $subName$output
done
But while doing echo of 11th column, its trimming the extra spaces which leads to mismatch in count of character. Is there any way to echo without trimming spaces ?
Actual output
10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR MHIN|||0000000010000.00|6761857316|508998|6011|GL
Expected Output
10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR|MH|IN|0000000010000.00|6761857316|508998|6011|GL
linux bash
Comments are not for extended discussion; this conversation has been moved to chat.
– Bhargav Rao♦
Jan 3 at 9:08
add a comment |
I have a file consisting of multiple rows like this
10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GRCHEMBUR MHIN|0000000010000.00|6761857316|508998|6011|GL
I have to split and replace the column 11 into 4 different columns using the count of character.
This is the 11th column containing extra spaces also.
SHOP NO.5,6,7 RUNWAL GRCHEMBUR MHIN
This is I have done
ls *.txt *.TXT| while read line
do
subName="$(cut -d'.' -f1 <<<"$line")"
awk -F"|" '{ "echo -n "$11" | cut -c1-23" | getline ton;
"echo -n "$11" | cut -c24-36" | getline city;
"echo -n "$11" | cut -c37-38" | getline state;
"echo -n "$11" | cut -c39-40" | getline country;
$11=ton"|"city"|"state"|"country; print $0
}' OFS="|" $line > $subName$output
done
But while doing echo of 11th column, its trimming the extra spaces which leads to mismatch in count of character. Is there any way to echo without trimming spaces ?
Actual output
10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR MHIN|||0000000010000.00|6761857316|508998|6011|GL
Expected Output
10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR|MH|IN|0000000010000.00|6761857316|508998|6011|GL
linux bash
I have a file consisting of multiple rows like this
10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GRCHEMBUR MHIN|0000000010000.00|6761857316|508998|6011|GL
I have to split and replace the column 11 into 4 different columns using the count of character.
This is the 11th column containing extra spaces also.
SHOP NO.5,6,7 RUNWAL GRCHEMBUR MHIN
This is I have done
ls *.txt *.TXT| while read line
do
subName="$(cut -d'.' -f1 <<<"$line")"
awk -F"|" '{ "echo -n "$11" | cut -c1-23" | getline ton;
"echo -n "$11" | cut -c24-36" | getline city;
"echo -n "$11" | cut -c37-38" | getline state;
"echo -n "$11" | cut -c39-40" | getline country;
$11=ton"|"city"|"state"|"country; print $0
}' OFS="|" $line > $subName$output
done
But while doing echo of 11th column, its trimming the extra spaces which leads to mismatch in count of character. Is there any way to echo without trimming spaces ?
Actual output
10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR MHIN|||0000000010000.00|6761857316|508998|6011|GL
Expected Output
10|EQU000000001|12345678|3456||EOMCO042|EOMCO042|31DEC2018|16:51:17|31DEC2018|SHOP NO.5,6,7 RUNWAL GR|CHEMBUR|MH|IN|0000000010000.00|6761857316|508998|6011|GL
linux bash
linux bash
edited Jan 2 at 16:26
Muddassir Rahman
asked Jan 2 at 15:56
Muddassir RahmanMuddassir Rahman
235110
235110
Comments are not for extended discussion; this conversation has been moved to chat.
– Bhargav Rao♦
Jan 3 at 9:08
add a comment |
Comments are not for extended discussion; this conversation has been moved to chat.
– Bhargav Rao♦
Jan 3 at 9:08
Comments are not for extended discussion; this conversation has been moved to chat.
– Bhargav Rao♦
Jan 3 at 9:08
Comments are not for extended discussion; this conversation has been moved to chat.
– Bhargav Rao♦
Jan 3 at 9:08
add a comment |
2 Answers
2
active
oldest
votes
A pure-bash implementation of all this logic
#!/usr/bin/env bash
shopt -s nocaseglob extglob
for f in *.txt; do
subName=${f%.*}
while IFS='|' read -r -a fields; do
location=${fields[10]}
ton=${location:0:23}; ton=${ton%%+([[:space:]])}
city=${location:23:12}; city=${city%%+([[:space:]])}
state=${location:36:2}
country=${location:38:2}
fields[10]="$ton|$city|$state|$country"
printf -v out '%s|' "${fields[@]}"
printf '%sn' "${out:0:$(( ${#out} - 1 ))}"
done <"$f" >"$subName.out"
done
It's slower (if I did this well, by about a factor of 10) than pure awk would be, but much faster than the awk/shell combination proposed in the question.
Going into the constructs used:
- All the
${varname%...}
and related constructs are parameter expansion. The specific${varname%pattern}
construct removes the shortest possible match forpattern
from the value invarname
, or the longest match if%
is replaced with%%
. - Using
extglob
enables extended globbing syntax, such as+([[:space:]])
, which is equivalent to the regex syntax[[:space:]]+
.
add a comment |
The least annoying way to code this that I've found so far is:
perl -F'|' -lane '$F[10] = join "|", unpack "a23 A13 a2 a2", $F[10]; print join "|", @F'
It's fairly straightforward:
- Iterate over lines of input; split each line on
|
and put the fields in@F
. - For the 11th field (
$F[10]
), split it into fixed-width subfields usingunpack
(and trim trailing spaces from the second field (A
instead ofa
)). - Reassemble subfields by joining with
|
. - Reassemble the whole line by joining with
|
and printing it.
I haven't benchmarked it in any way, but it's likely much faster than the original code that spawns multiple shell and cut processes per input line because it's all done in one process.
A complete solution would wrap it in a shell loop:
for file in *.txt *.TXT; do
outfile="${file%.*}$output"
perl -F'|' -lane '...' "$file" > "$outfile"
done
Or if you don't need to trim the .txt
part (and you don't have too many files to fit on the command line):
perl -i.out -F'|' -lane '...' *.txt *.TXT
This simply places the output for each input file foo.txt
in foo.txt.out
.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54009349%2fecho-without-trimming-the-space-in-awk-command%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
A pure-bash implementation of all this logic
#!/usr/bin/env bash
shopt -s nocaseglob extglob
for f in *.txt; do
subName=${f%.*}
while IFS='|' read -r -a fields; do
location=${fields[10]}
ton=${location:0:23}; ton=${ton%%+([[:space:]])}
city=${location:23:12}; city=${city%%+([[:space:]])}
state=${location:36:2}
country=${location:38:2}
fields[10]="$ton|$city|$state|$country"
printf -v out '%s|' "${fields[@]}"
printf '%sn' "${out:0:$(( ${#out} - 1 ))}"
done <"$f" >"$subName.out"
done
It's slower (if I did this well, by about a factor of 10) than pure awk would be, but much faster than the awk/shell combination proposed in the question.
Going into the constructs used:
- All the
${varname%...}
and related constructs are parameter expansion. The specific${varname%pattern}
construct removes the shortest possible match forpattern
from the value invarname
, or the longest match if%
is replaced with%%
. - Using
extglob
enables extended globbing syntax, such as+([[:space:]])
, which is equivalent to the regex syntax[[:space:]]+
.
add a comment |
A pure-bash implementation of all this logic
#!/usr/bin/env bash
shopt -s nocaseglob extglob
for f in *.txt; do
subName=${f%.*}
while IFS='|' read -r -a fields; do
location=${fields[10]}
ton=${location:0:23}; ton=${ton%%+([[:space:]])}
city=${location:23:12}; city=${city%%+([[:space:]])}
state=${location:36:2}
country=${location:38:2}
fields[10]="$ton|$city|$state|$country"
printf -v out '%s|' "${fields[@]}"
printf '%sn' "${out:0:$(( ${#out} - 1 ))}"
done <"$f" >"$subName.out"
done
It's slower (if I did this well, by about a factor of 10) than pure awk would be, but much faster than the awk/shell combination proposed in the question.
Going into the constructs used:
- All the
${varname%...}
and related constructs are parameter expansion. The specific${varname%pattern}
construct removes the shortest possible match forpattern
from the value invarname
, or the longest match if%
is replaced with%%
. - Using
extglob
enables extended globbing syntax, such as+([[:space:]])
, which is equivalent to the regex syntax[[:space:]]+
.
add a comment |
A pure-bash implementation of all this logic
#!/usr/bin/env bash
shopt -s nocaseglob extglob
for f in *.txt; do
subName=${f%.*}
while IFS='|' read -r -a fields; do
location=${fields[10]}
ton=${location:0:23}; ton=${ton%%+([[:space:]])}
city=${location:23:12}; city=${city%%+([[:space:]])}
state=${location:36:2}
country=${location:38:2}
fields[10]="$ton|$city|$state|$country"
printf -v out '%s|' "${fields[@]}"
printf '%sn' "${out:0:$(( ${#out} - 1 ))}"
done <"$f" >"$subName.out"
done
It's slower (if I did this well, by about a factor of 10) than pure awk would be, but much faster than the awk/shell combination proposed in the question.
Going into the constructs used:
- All the
${varname%...}
and related constructs are parameter expansion. The specific${varname%pattern}
construct removes the shortest possible match forpattern
from the value invarname
, or the longest match if%
is replaced with%%
. - Using
extglob
enables extended globbing syntax, such as+([[:space:]])
, which is equivalent to the regex syntax[[:space:]]+
.
A pure-bash implementation of all this logic
#!/usr/bin/env bash
shopt -s nocaseglob extglob
for f in *.txt; do
subName=${f%.*}
while IFS='|' read -r -a fields; do
location=${fields[10]}
ton=${location:0:23}; ton=${ton%%+([[:space:]])}
city=${location:23:12}; city=${city%%+([[:space:]])}
state=${location:36:2}
country=${location:38:2}
fields[10]="$ton|$city|$state|$country"
printf -v out '%s|' "${fields[@]}"
printf '%sn' "${out:0:$(( ${#out} - 1 ))}"
done <"$f" >"$subName.out"
done
It's slower (if I did this well, by about a factor of 10) than pure awk would be, but much faster than the awk/shell combination proposed in the question.
Going into the constructs used:
- All the
${varname%...}
and related constructs are parameter expansion. The specific${varname%pattern}
construct removes the shortest possible match forpattern
from the value invarname
, or the longest match if%
is replaced with%%
. - Using
extglob
enables extended globbing syntax, such as+([[:space:]])
, which is equivalent to the regex syntax[[:space:]]+
.
edited Jan 2 at 16:41
answered Jan 2 at 16:11
Charles DuffyCharles Duffy
179k25204259
179k25204259
add a comment |
add a comment |
The least annoying way to code this that I've found so far is:
perl -F'|' -lane '$F[10] = join "|", unpack "a23 A13 a2 a2", $F[10]; print join "|", @F'
It's fairly straightforward:
- Iterate over lines of input; split each line on
|
and put the fields in@F
. - For the 11th field (
$F[10]
), split it into fixed-width subfields usingunpack
(and trim trailing spaces from the second field (A
instead ofa
)). - Reassemble subfields by joining with
|
. - Reassemble the whole line by joining with
|
and printing it.
I haven't benchmarked it in any way, but it's likely much faster than the original code that spawns multiple shell and cut processes per input line because it's all done in one process.
A complete solution would wrap it in a shell loop:
for file in *.txt *.TXT; do
outfile="${file%.*}$output"
perl -F'|' -lane '...' "$file" > "$outfile"
done
Or if you don't need to trim the .txt
part (and you don't have too many files to fit on the command line):
perl -i.out -F'|' -lane '...' *.txt *.TXT
This simply places the output for each input file foo.txt
in foo.txt.out
.
add a comment |
The least annoying way to code this that I've found so far is:
perl -F'|' -lane '$F[10] = join "|", unpack "a23 A13 a2 a2", $F[10]; print join "|", @F'
It's fairly straightforward:
- Iterate over lines of input; split each line on
|
and put the fields in@F
. - For the 11th field (
$F[10]
), split it into fixed-width subfields usingunpack
(and trim trailing spaces from the second field (A
instead ofa
)). - Reassemble subfields by joining with
|
. - Reassemble the whole line by joining with
|
and printing it.
I haven't benchmarked it in any way, but it's likely much faster than the original code that spawns multiple shell and cut processes per input line because it's all done in one process.
A complete solution would wrap it in a shell loop:
for file in *.txt *.TXT; do
outfile="${file%.*}$output"
perl -F'|' -lane '...' "$file" > "$outfile"
done
Or if you don't need to trim the .txt
part (and you don't have too many files to fit on the command line):
perl -i.out -F'|' -lane '...' *.txt *.TXT
This simply places the output for each input file foo.txt
in foo.txt.out
.
add a comment |
The least annoying way to code this that I've found so far is:
perl -F'|' -lane '$F[10] = join "|", unpack "a23 A13 a2 a2", $F[10]; print join "|", @F'
It's fairly straightforward:
- Iterate over lines of input; split each line on
|
and put the fields in@F
. - For the 11th field (
$F[10]
), split it into fixed-width subfields usingunpack
(and trim trailing spaces from the second field (A
instead ofa
)). - Reassemble subfields by joining with
|
. - Reassemble the whole line by joining with
|
and printing it.
I haven't benchmarked it in any way, but it's likely much faster than the original code that spawns multiple shell and cut processes per input line because it's all done in one process.
A complete solution would wrap it in a shell loop:
for file in *.txt *.TXT; do
outfile="${file%.*}$output"
perl -F'|' -lane '...' "$file" > "$outfile"
done
Or if you don't need to trim the .txt
part (and you don't have too many files to fit on the command line):
perl -i.out -F'|' -lane '...' *.txt *.TXT
This simply places the output for each input file foo.txt
in foo.txt.out
.
The least annoying way to code this that I've found so far is:
perl -F'|' -lane '$F[10] = join "|", unpack "a23 A13 a2 a2", $F[10]; print join "|", @F'
It's fairly straightforward:
- Iterate over lines of input; split each line on
|
and put the fields in@F
. - For the 11th field (
$F[10]
), split it into fixed-width subfields usingunpack
(and trim trailing spaces from the second field (A
instead ofa
)). - Reassemble subfields by joining with
|
. - Reassemble the whole line by joining with
|
and printing it.
I haven't benchmarked it in any way, but it's likely much faster than the original code that spawns multiple shell and cut processes per input line because it's all done in one process.
A complete solution would wrap it in a shell loop:
for file in *.txt *.TXT; do
outfile="${file%.*}$output"
perl -F'|' -lane '...' "$file" > "$outfile"
done
Or if you don't need to trim the .txt
part (and you don't have too many files to fit on the command line):
perl -i.out -F'|' -lane '...' *.txt *.TXT
This simply places the output for each input file foo.txt
in foo.txt.out
.
edited Jan 2 at 16:42
answered Jan 2 at 16:32
melpomenemelpomene
61.7k54994
61.7k54994
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54009349%2fecho-without-trimming-the-space-in-awk-command%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
3o4a rbKUmCy4 pqsn4z8 38uNSqwmq7xgJiuGVCq su5kg3QswKeAWJZ p,NuVxrHFh1FADC8EblaX,6Xk oYVD 8j,y,uPsfQLMV
Comments are not for extended discussion; this conversation has been moved to chat.
– Bhargav Rao♦
Jan 3 at 9:08