sed -i command for in-place editing to work with both GNU sed and BSD/OSX

Question

I've got a makefile (developed for gmake on Linux) that I'm attempting to port to MacOS, but it seems like sed doesn't want to cooperate. What I do is use GCC to autogenerate dependency files, and then tweak them a bit using sed. The relevant portion of the makefile:

$(OBJ_DIR)/%.d: $(SRC_DIR)/%.cpp
  $(CPPC) -MM -MD $< -o $@
  sed -i 's|\(.*\)\.o:|$(OBJ_DIR)/\1.o $(OBJ_DIR)/\1.d $(TEST_OBJ_DIR)/\1_utest.o:|' $@

While this runs with no trouble under GNU/Linux, I get errors like the following when attempting to build on MacOS:

sed: 1: "test/obj/equipmentConta ...": undefined label 'est/obj/equipmentContainer_utest.d'
sed: 1: "test/obj/dice_utest.d": undefined label 'est/obj/dice_utest.d'
sed: 1: "test/obj/color-string_u ...": undefined label 'est/obj/color-string_utest.d'

It would seem like sed is chopping off a character, but I can't see the solution.

Answer 1

OS X sed handles the -i argument differently to the Linux version.

You can generate a command that might "work" for both by adding -e in this way:

#      vv
sed -i -e 's|\(.*\)\.o:|$(OBJ_DIR)/\1.o $(OBJ_DIR)/\1.d $(TEST_OBJ_DIR)/\1_utest.o:|' $@

OS X sed -i interprets the next thing after the -i as a file extension for a backup copy of the in-place edit. (The Linux version only does this if there is no space between the -i and the extension.) Obviously a side affect of using this is that you will get a backup file with -e as an extension, which you may not want. Please refer to other answers to this question for more details, and cleaner approaches that can be used instead.

The behaviour you see is because OS X sed consumes the s||| as the extension (!) then interprets the next argument as a command - in this case it begins with t, which sed recognizes as a branch-to-label command expecting the target label as an argument - hence the error you see.

If you create a file test you can reproduce the error:

$ sed -i 's|x|y|' test
sed: 1: "test": undefined label 'est'

Answer 2

I came across this issue as well and thought of the following solution:

darwin=false;
case "`uname`" in
  Darwin*) darwin=true ;;
esac

if $darwin; then
  sedi="/usr/bin/sed -i ''"
else
  sedi="sed -i"
fi

$sedi 's/foo/bar/' /home/foobar/bar

Works for me ;-), YMMV

I work in a multi-OS team where ppl build on Windows, Linux and OS X. Some OS X users complained because they got another error - they had the GNU port of sed installed so I had to specify the full path.

Answer 3

This isn't quite an answer to the question, but one can get linux-equivalent behavior through

brew install gnu-sed

# Add to .bashrc / .zshrc
export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"

(previously there was a --with-default-names option to brew install gnu-sed but that has recently been removed)

Answer 4

Actually, doing

sed -i -e "s/blah/blah/" files

doesn't do what you expect in MacOS either. Instead it creates backup files with -e extension.

The proper command for MacOS is

sed -i "" -e "s/blah/blah/" files

On Linux, remove the space between -i and "" (see related answer)

sed -i"" -e "s/blah/blah/" files

Answer 5

martin clayton's helpful answer provides a good explanation of the problem^[1], but his solution - as he states - has a potentially unwanted side effect.

Here are side-effect-free solutions:

Caveat: Solving the -i syntax problem alone, as below, may not be enough, because there are many other differences between GNU sed and BSD/macOS sed (for a comprehensive discussion, see this answer of mine).

---

Workaround with -i: Create a backup file temporarily, then clean it up:

With a non-empty suffix (backup-file filename extension) option-argument (a value that is not the empty string), you can use -i in a way that works with both BSD/macOS sed and GNU sed, by directly appending the suffix to the -i option.

This can be utilized to create a backup file temporarily that you can clean up right away:

sed -i.bak 's/foo/bar/' file && rm file.bak

Obviously, if you do want to keep the backup, simply omit the && rm file.bak part.

---

Workaround that is POSIX-compliant, using a temporary file and mv:

If only a single file is to be edited in-place, the -i option can be bypassed to avoid the incompatibility.

If you restrict your sed script and other options to POSIX-compliant features, the following is a fully portable solution (note that -i is not POSIX-compliant).

sed 's/foo/bar' file > /tmp/file.$$ && mv /tmp/file.$$ file

• This command simply writes the modifications to a temporary file and, if the sed command succeeds (&&), replaces the original file with the temporary one.

  • If you do want to keep the original file as a backup, add another mv command that renames the original first.

• Caveat: Fundamentally, this is what -i does too, except that it tries to preserve permissions and extended attributes (macOS) of the original file; however, if the original file is a symlink, both this solution and -i will replace the symlink with a regular file.
  ^{See the bottom half of this answer of mine for details on how -i works.}

---

^{[1] For a more in-depth explanation, see this answer of mine.}

Answer 6

I've corrected the solution posted by @thecarpy:

Here's a proper cross-platform solution for sed -i:

sedi() {
  case $(uname) in
    Darwin*) sedi=('-i' '') ;;
    *) sedi='-i' ;;
  esac

  LC_ALL=C sed "${sedi[@]}" "$@"
}

Answer 7

I avoid using sed -i when writing scripts and i came up with simple solution:

printf '%s' "$(sed 's/foo/bar' file)" > file

much compatible and is POSIX-compliant. It is doing pretty much the same as sed -i, but this one does not create temp files, it directly redirect the changes to file.

As a noob idk what's the cons of doing this, the only matters is "It works"

Answer 8

The currently accepted answer is flawed in two very important ways.

1. With BSD sed (the OSX version), the -e option is interpreted as a file extension and therefore creates a backup file with a -e extension.

2. Testing for the darwin kernel as suggested is not a reliable approach to a cross platform solution since GNU or BSD sed could be present on any number of systems.

A much more reliable test would be to simply test for the --version option which is only found in the GNU version of sed.

sed --version >/dev/null 2>&1

Once the correct version of sed is determined, we can then execute the command in its proper syntax.

GNU sed syntax for -i option:

sed -i -- "$@"

BSD sed syntax for -i option:

sed -i "" "$@"

Finally put it all together in a cross platform function to execute an in place edit sed commend:

sedi () {
    sed --version >/dev/null 2>&1 && sed -i -- "$@" || sed -i "" "$@"
}

Example usage:

sedi 's/old/new/g' 'some_file.txt'

This solution has been tested on OSX, Ubuntu, Freebsd, Cygwin, CentOS, Red Hat Enterprise, & Msys.