will io direction operation lock the file?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
i have a growing nginx log file about 20G already, and i wish to rotate it.
1, i mv
the old log file to a new log file
2, i do > old_log_file.log
to truncate the old log file in about 2~3 seconds
if there's a lock(write lock?) on the old log file when i doing the truncating(about 2~3 seconds)?
at that 2~3s period, nginx returns 502 for waiting to append logs to old log file until lock released?
thank you for explaining.
linux redirect io truncate
|
show 1 more comment
i have a growing nginx log file about 20G already, and i wish to rotate it.
1, i mv
the old log file to a new log file
2, i do > old_log_file.log
to truncate the old log file in about 2~3 seconds
if there's a lock(write lock?) on the old log file when i doing the truncating(about 2~3 seconds)?
at that 2~3s period, nginx returns 502 for waiting to append logs to old log file until lock released?
thank you for explaining.
linux redirect io truncate
mv old_log_file.log new_log_file.log && touch old_log_file.log
– David C. Rankin
Jan 5 at 9:02
i don't think your comand would work, becausemv
does not change theinode
of old log file. Andnginx
would still log into the "new_log_file" because of theinode
. And you create a new log file named as olg log file, but theinode
changed, nginx wouldn't write into it untilnginx -s reload
the config file.
– touchstone
Jan 7 at 6:08
If that is hownginx
associates the log, then you may well be correct. Then why notcp -a old_log_file.log new_log_file.log && :> old_log_file.log
?
– David C. Rankin
Jan 7 at 6:17
i tried 2 truncating methods:>
and 'logrotate', they all caused the nginx output 502 -- very short period, after then 200. andlogrotate
's handling procedure is as you said truncate aftercp
. now i have no idea what to debug this bug...
– touchstone
Jan 7 at 6:22
It is not a bug, it is a feature.
– Basile Starynkevitch
Jan 7 at 8:23
|
show 1 more comment
i have a growing nginx log file about 20G already, and i wish to rotate it.
1, i mv
the old log file to a new log file
2, i do > old_log_file.log
to truncate the old log file in about 2~3 seconds
if there's a lock(write lock?) on the old log file when i doing the truncating(about 2~3 seconds)?
at that 2~3s period, nginx returns 502 for waiting to append logs to old log file until lock released?
thank you for explaining.
linux redirect io truncate
i have a growing nginx log file about 20G already, and i wish to rotate it.
1, i mv
the old log file to a new log file
2, i do > old_log_file.log
to truncate the old log file in about 2~3 seconds
if there's a lock(write lock?) on the old log file when i doing the truncating(about 2~3 seconds)?
at that 2~3s period, nginx returns 502 for waiting to append logs to old log file until lock released?
thank you for explaining.
linux redirect io truncate
linux redirect io truncate
asked Jan 4 at 6:21
touchstonetouchstone
16718
16718
mv old_log_file.log new_log_file.log && touch old_log_file.log
– David C. Rankin
Jan 5 at 9:02
i don't think your comand would work, becausemv
does not change theinode
of old log file. Andnginx
would still log into the "new_log_file" because of theinode
. And you create a new log file named as olg log file, but theinode
changed, nginx wouldn't write into it untilnginx -s reload
the config file.
– touchstone
Jan 7 at 6:08
If that is hownginx
associates the log, then you may well be correct. Then why notcp -a old_log_file.log new_log_file.log && :> old_log_file.log
?
– David C. Rankin
Jan 7 at 6:17
i tried 2 truncating methods:>
and 'logrotate', they all caused the nginx output 502 -- very short period, after then 200. andlogrotate
's handling procedure is as you said truncate aftercp
. now i have no idea what to debug this bug...
– touchstone
Jan 7 at 6:22
It is not a bug, it is a feature.
– Basile Starynkevitch
Jan 7 at 8:23
|
show 1 more comment
mv old_log_file.log new_log_file.log && touch old_log_file.log
– David C. Rankin
Jan 5 at 9:02
i don't think your comand would work, becausemv
does not change theinode
of old log file. Andnginx
would still log into the "new_log_file" because of theinode
. And you create a new log file named as olg log file, but theinode
changed, nginx wouldn't write into it untilnginx -s reload
the config file.
– touchstone
Jan 7 at 6:08
If that is hownginx
associates the log, then you may well be correct. Then why notcp -a old_log_file.log new_log_file.log && :> old_log_file.log
?
– David C. Rankin
Jan 7 at 6:17
i tried 2 truncating methods:>
and 'logrotate', they all caused the nginx output 502 -- very short period, after then 200. andlogrotate
's handling procedure is as you said truncate aftercp
. now i have no idea what to debug this bug...
– touchstone
Jan 7 at 6:22
It is not a bug, it is a feature.
– Basile Starynkevitch
Jan 7 at 8:23
mv old_log_file.log new_log_file.log && touch old_log_file.log
– David C. Rankin
Jan 5 at 9:02
mv old_log_file.log new_log_file.log && touch old_log_file.log
– David C. Rankin
Jan 5 at 9:02
i don't think your comand would work, because
mv
does not change the inode
of old log file. And nginx
would still log into the "new_log_file" because of the inode
. And you create a new log file named as olg log file, but the inode
changed, nginx wouldn't write into it until nginx -s reload
the config file.– touchstone
Jan 7 at 6:08
i don't think your comand would work, because
mv
does not change the inode
of old log file. And nginx
would still log into the "new_log_file" because of the inode
. And you create a new log file named as olg log file, but the inode
changed, nginx wouldn't write into it until nginx -s reload
the config file.– touchstone
Jan 7 at 6:08
If that is how
nginx
associates the log, then you may well be correct. Then why not cp -a old_log_file.log new_log_file.log && :> old_log_file.log
?– David C. Rankin
Jan 7 at 6:17
If that is how
nginx
associates the log, then you may well be correct. Then why not cp -a old_log_file.log new_log_file.log && :> old_log_file.log
?– David C. Rankin
Jan 7 at 6:17
i tried 2 truncating methods:
>
and 'logrotate', they all caused the nginx output 502 -- very short period, after then 200. and logrotate
's handling procedure is as you said truncate after cp
. now i have no idea what to debug this bug...– touchstone
Jan 7 at 6:22
i tried 2 truncating methods:
>
and 'logrotate', they all caused the nginx output 502 -- very short period, after then 200. and logrotate
's handling procedure is as you said truncate after cp
. now i have no idea what to debug this bug...– touchstone
Jan 7 at 6:22
It is not a bug, it is a feature.
– Basile Starynkevitch
Jan 7 at 8:23
It is not a bug, it is a feature.
– Basile Starynkevitch
Jan 7 at 8:23
|
show 1 more comment
2 Answers
2
active
oldest
votes
On Linux, there is (almost) no mandatory file locks (more precisely, there used to be some mandatory locking feature in the kernel, but it is deprecated and you really should avoid using it). File locking happens with flock(2) or lockf(3) and is advisory and should be explicit (e.g. with flock(1) command, or some program calling flock
or lockf
).
So every locking related to files is practically a convention between all the software using that file (and mv(1) or the redirection by your shell don't use file locking).
Remember that a file on Linux is mostly an i-node (see inode(7)) which could have zero, one or several file paths (see path_resolution(7) and be aware of link(2), rename(2), unlink(2)) and used thru some file descriptor. Read ALP (and perhaps Operating Systems: Three Easy Pieces) for more.
No file locking happens in the scenario of your question (and the i-nodes and file descriptors involved are independent).
Consider using logrotate(8).
Some software provide a way to reload their configuration and re-open log files. You should read the documentation of your nginx.
logrotate
's source code shows that it firstcp
the old one then truncate the old one,inode
doesn't change. and actually i usedlogrotate
after>
causes 502 issue, but the issues still exist, then i asked this question. sorry for my english.
– touchstone
Jan 7 at 5:57
logrotate
andkill -USR1 nginx.pid
which would reload the config file, all i tried does not solve the issue.
– touchstone
Jan 7 at 6:04
fine. i learned that ">
" doesn't lock the file even though 2~3s' truncating at least. thank you a lot, i would keep looking for answers for my 502.
– touchstone
Jan 7 at 6:15
Reason in terms of i-nodes and file descriptors. You'll understand better what happens in your computer
– Basile Starynkevitch
Jan 7 at 6:36
maybe, the file offset is decreasing quickly as>
is truncating and at the same time nginx is appending logs with a shrinking offset (i don't know how to describe a unstable offset)
– touchstone
Jan 7 at 8:22
add a comment |
It depends on application if it locks the file. Application that generates this log file must have option to clear log file. One example is in editor like vim file can be externally modified while it is still open in editor.
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54033940%2fwill-io-direction-operation-lock-the-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
On Linux, there is (almost) no mandatory file locks (more precisely, there used to be some mandatory locking feature in the kernel, but it is deprecated and you really should avoid using it). File locking happens with flock(2) or lockf(3) and is advisory and should be explicit (e.g. with flock(1) command, or some program calling flock
or lockf
).
So every locking related to files is practically a convention between all the software using that file (and mv(1) or the redirection by your shell don't use file locking).
Remember that a file on Linux is mostly an i-node (see inode(7)) which could have zero, one or several file paths (see path_resolution(7) and be aware of link(2), rename(2), unlink(2)) and used thru some file descriptor. Read ALP (and perhaps Operating Systems: Three Easy Pieces) for more.
No file locking happens in the scenario of your question (and the i-nodes and file descriptors involved are independent).
Consider using logrotate(8).
Some software provide a way to reload their configuration and re-open log files. You should read the documentation of your nginx.
logrotate
's source code shows that it firstcp
the old one then truncate the old one,inode
doesn't change. and actually i usedlogrotate
after>
causes 502 issue, but the issues still exist, then i asked this question. sorry for my english.
– touchstone
Jan 7 at 5:57
logrotate
andkill -USR1 nginx.pid
which would reload the config file, all i tried does not solve the issue.
– touchstone
Jan 7 at 6:04
fine. i learned that ">
" doesn't lock the file even though 2~3s' truncating at least. thank you a lot, i would keep looking for answers for my 502.
– touchstone
Jan 7 at 6:15
Reason in terms of i-nodes and file descriptors. You'll understand better what happens in your computer
– Basile Starynkevitch
Jan 7 at 6:36
maybe, the file offset is decreasing quickly as>
is truncating and at the same time nginx is appending logs with a shrinking offset (i don't know how to describe a unstable offset)
– touchstone
Jan 7 at 8:22
add a comment |
On Linux, there is (almost) no mandatory file locks (more precisely, there used to be some mandatory locking feature in the kernel, but it is deprecated and you really should avoid using it). File locking happens with flock(2) or lockf(3) and is advisory and should be explicit (e.g. with flock(1) command, or some program calling flock
or lockf
).
So every locking related to files is practically a convention between all the software using that file (and mv(1) or the redirection by your shell don't use file locking).
Remember that a file on Linux is mostly an i-node (see inode(7)) which could have zero, one or several file paths (see path_resolution(7) and be aware of link(2), rename(2), unlink(2)) and used thru some file descriptor. Read ALP (and perhaps Operating Systems: Three Easy Pieces) for more.
No file locking happens in the scenario of your question (and the i-nodes and file descriptors involved are independent).
Consider using logrotate(8).
Some software provide a way to reload their configuration and re-open log files. You should read the documentation of your nginx.
logrotate
's source code shows that it firstcp
the old one then truncate the old one,inode
doesn't change. and actually i usedlogrotate
after>
causes 502 issue, but the issues still exist, then i asked this question. sorry for my english.
– touchstone
Jan 7 at 5:57
logrotate
andkill -USR1 nginx.pid
which would reload the config file, all i tried does not solve the issue.
– touchstone
Jan 7 at 6:04
fine. i learned that ">
" doesn't lock the file even though 2~3s' truncating at least. thank you a lot, i would keep looking for answers for my 502.
– touchstone
Jan 7 at 6:15
Reason in terms of i-nodes and file descriptors. You'll understand better what happens in your computer
– Basile Starynkevitch
Jan 7 at 6:36
maybe, the file offset is decreasing quickly as>
is truncating and at the same time nginx is appending logs with a shrinking offset (i don't know how to describe a unstable offset)
– touchstone
Jan 7 at 8:22
add a comment |
On Linux, there is (almost) no mandatory file locks (more precisely, there used to be some mandatory locking feature in the kernel, but it is deprecated and you really should avoid using it). File locking happens with flock(2) or lockf(3) and is advisory and should be explicit (e.g. with flock(1) command, or some program calling flock
or lockf
).
So every locking related to files is practically a convention between all the software using that file (and mv(1) or the redirection by your shell don't use file locking).
Remember that a file on Linux is mostly an i-node (see inode(7)) which could have zero, one or several file paths (see path_resolution(7) and be aware of link(2), rename(2), unlink(2)) and used thru some file descriptor. Read ALP (and perhaps Operating Systems: Three Easy Pieces) for more.
No file locking happens in the scenario of your question (and the i-nodes and file descriptors involved are independent).
Consider using logrotate(8).
Some software provide a way to reload their configuration and re-open log files. You should read the documentation of your nginx.
On Linux, there is (almost) no mandatory file locks (more precisely, there used to be some mandatory locking feature in the kernel, but it is deprecated and you really should avoid using it). File locking happens with flock(2) or lockf(3) and is advisory and should be explicit (e.g. with flock(1) command, or some program calling flock
or lockf
).
So every locking related to files is practically a convention between all the software using that file (and mv(1) or the redirection by your shell don't use file locking).
Remember that a file on Linux is mostly an i-node (see inode(7)) which could have zero, one or several file paths (see path_resolution(7) and be aware of link(2), rename(2), unlink(2)) and used thru some file descriptor. Read ALP (and perhaps Operating Systems: Three Easy Pieces) for more.
No file locking happens in the scenario of your question (and the i-nodes and file descriptors involved are independent).
Consider using logrotate(8).
Some software provide a way to reload their configuration and re-open log files. You should read the documentation of your nginx.
edited Jan 5 at 17:24
answered Jan 5 at 8:38
Basile StarynkevitchBasile Starynkevitch
180k13174375
180k13174375
logrotate
's source code shows that it firstcp
the old one then truncate the old one,inode
doesn't change. and actually i usedlogrotate
after>
causes 502 issue, but the issues still exist, then i asked this question. sorry for my english.
– touchstone
Jan 7 at 5:57
logrotate
andkill -USR1 nginx.pid
which would reload the config file, all i tried does not solve the issue.
– touchstone
Jan 7 at 6:04
fine. i learned that ">
" doesn't lock the file even though 2~3s' truncating at least. thank you a lot, i would keep looking for answers for my 502.
– touchstone
Jan 7 at 6:15
Reason in terms of i-nodes and file descriptors. You'll understand better what happens in your computer
– Basile Starynkevitch
Jan 7 at 6:36
maybe, the file offset is decreasing quickly as>
is truncating and at the same time nginx is appending logs with a shrinking offset (i don't know how to describe a unstable offset)
– touchstone
Jan 7 at 8:22
add a comment |
logrotate
's source code shows that it firstcp
the old one then truncate the old one,inode
doesn't change. and actually i usedlogrotate
after>
causes 502 issue, but the issues still exist, then i asked this question. sorry for my english.
– touchstone
Jan 7 at 5:57
logrotate
andkill -USR1 nginx.pid
which would reload the config file, all i tried does not solve the issue.
– touchstone
Jan 7 at 6:04
fine. i learned that ">
" doesn't lock the file even though 2~3s' truncating at least. thank you a lot, i would keep looking for answers for my 502.
– touchstone
Jan 7 at 6:15
Reason in terms of i-nodes and file descriptors. You'll understand better what happens in your computer
– Basile Starynkevitch
Jan 7 at 6:36
maybe, the file offset is decreasing quickly as>
is truncating and at the same time nginx is appending logs with a shrinking offset (i don't know how to describe a unstable offset)
– touchstone
Jan 7 at 8:22
logrotate
's source code shows that it first cp
the old one then truncate the old one, inode
doesn't change. and actually i used logrotate
after >
causes 502 issue, but the issues still exist, then i asked this question. sorry for my english.– touchstone
Jan 7 at 5:57
logrotate
's source code shows that it first cp
the old one then truncate the old one, inode
doesn't change. and actually i used logrotate
after >
causes 502 issue, but the issues still exist, then i asked this question. sorry for my english.– touchstone
Jan 7 at 5:57
logrotate
and kill -USR1 nginx.pid
which would reload the config file, all i tried does not solve the issue.– touchstone
Jan 7 at 6:04
logrotate
and kill -USR1 nginx.pid
which would reload the config file, all i tried does not solve the issue.– touchstone
Jan 7 at 6:04
fine. i learned that "
>
" doesn't lock the file even though 2~3s' truncating at least. thank you a lot, i would keep looking for answers for my 502.– touchstone
Jan 7 at 6:15
fine. i learned that "
>
" doesn't lock the file even though 2~3s' truncating at least. thank you a lot, i would keep looking for answers for my 502.– touchstone
Jan 7 at 6:15
Reason in terms of i-nodes and file descriptors. You'll understand better what happens in your computer
– Basile Starynkevitch
Jan 7 at 6:36
Reason in terms of i-nodes and file descriptors. You'll understand better what happens in your computer
– Basile Starynkevitch
Jan 7 at 6:36
maybe, the file offset is decreasing quickly as
>
is truncating and at the same time nginx is appending logs with a shrinking offset (i don't know how to describe a unstable offset)– touchstone
Jan 7 at 8:22
maybe, the file offset is decreasing quickly as
>
is truncating and at the same time nginx is appending logs with a shrinking offset (i don't know how to describe a unstable offset)– touchstone
Jan 7 at 8:22
add a comment |
It depends on application if it locks the file. Application that generates this log file must have option to clear log file. One example is in editor like vim file can be externally modified while it is still open in editor.
add a comment |
It depends on application if it locks the file. Application that generates this log file must have option to clear log file. One example is in editor like vim file can be externally modified while it is still open in editor.
add a comment |
It depends on application if it locks the file. Application that generates this log file must have option to clear log file. One example is in editor like vim file can be externally modified while it is still open in editor.
It depends on application if it locks the file. Application that generates this log file must have option to clear log file. One example is in editor like vim file can be externally modified while it is still open in editor.
answered Jan 5 at 8:36
anandanand
1337
1337
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54033940%2fwill-io-direction-operation-lock-the-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
mv old_log_file.log new_log_file.log && touch old_log_file.log
– David C. Rankin
Jan 5 at 9:02
i don't think your comand would work, because
mv
does not change theinode
of old log file. Andnginx
would still log into the "new_log_file" because of theinode
. And you create a new log file named as olg log file, but theinode
changed, nginx wouldn't write into it untilnginx -s reload
the config file.– touchstone
Jan 7 at 6:08
If that is how
nginx
associates the log, then you may well be correct. Then why notcp -a old_log_file.log new_log_file.log && :> old_log_file.log
?– David C. Rankin
Jan 7 at 6:17
i tried 2 truncating methods:
>
and 'logrotate', they all caused the nginx output 502 -- very short period, after then 200. andlogrotate
's handling procedure is as you said truncate aftercp
. now i have no idea what to debug this bug...– touchstone
Jan 7 at 6:22
It is not a bug, it is a feature.
– Basile Starynkevitch
Jan 7 at 8:23