Shell snippets

Some advanced shell stuff I need from time to time, perhaps this helps others as well.

• shell built-ins don’t have real manpages – they are documented in the manpage of their shell
• POSIX Shell Command Language
• Allow comments in interactive zsh shells with set -k (otherwise you might get some errors with the examples)

Enable “abort on error”

1

set -e

Abort shell scripts if a command fails. It doesn’t trigger in anything called from if, elif, while and until conditions, and only in the last component of && and || expressions (read the man page for details of your shell).

Shells usually accept -e as parameter too, like #!/bin/bash -e

Example 1

1

f() { set -e; echo "a"; false; echo "b"; }; (f)

This prints only the line with “a” – then the shell exits (f is called in a subshell so your real shell doesn’t close if you try this)

Example 2

1

f() { set -e; echo "a"; false; echo "b"; }; (f || echo "c")

This prints “a” and then “b” – set -e doesn’t have any effect as it is called from an || expression, it doesn’t exit on false, and echo "b" returns success. This time the shell doesn’t exit, so the subshell (…) could be removed.

So don’t use set -e in scripts where you call complex functions in conditionals – it probably doesn’t work like you want it to, but it is nice in small scripts to simplify error handling.

References

• POSIX set built-in
• bash set
• Part of the zshoptions “sh/ksh emulation set” (ERR_EXIT)

Quoting

Quoting is needed to deal with variables containing spaces (and similar special characters).

Quoting single arguments is usually easy, just put them in double quotes and everythings works fine, like this:

1
2

target="${1}"
cd "${target}"

You don’t need quotes to copy a variable (although I like to add them):

1
2

target=$1
cd "$target"

The curly braces are only needed in special cases like this:

1
2

text=abc
echo "${text}X"

To build large argument lists you need arrays:

Arrays

Not supported by POSIX sh, except for this feature: you can loop through all arguments like this:

1
2
3

for x in "$@"; do
  echo "$x"
done

bash and zsh support named arrays:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# print each parameter on a single line
pa() { local x; for x in "$@"; do echo "$x"; done; }

# create an array with zero elements
a=()
# append elements to an array
text="abc xyz"
a+=("$text" Z)

# expand each array entry to a separate parameter
pa "${a[@]}"

echo "Entries: ${#a[@]}"

If you need two dimensional arrays you have to use some way to “flatten” it – if the second dimension is fixed this is easy, otherwise you could store offset and length of each subarray in the flattened array.

Associative arrays are also supported:

1
2
3
4
5
6

# Need to tell the shell that "a" is an associatve array.
unset a; declare -A a

a[a]="abc"

echo "Entry 'a': ${a[a]}"

You can get a list of all keys with "${!t[@]}" in bash and "${(k)t[@]}" in zsh. The list of all entries works the same as in normal arrays.

Temporary files

If you create temporary stuff in your scripts, make sure you clean it up afterwards. Usually you should clean up even if errors happened, and a simple way to do this are EXIT traps.

1
2
3
4

tmpdir=$(mktemp --tmpdir -d myscriptname-XXXXXXX)
trap 'rm -rf "${tmpdir}"' EXIT

# now put all your stuff in "${tmpdir}/"

You can have only one exit trap – if you need a more dynamic hook system you have to build it yourself (register functions to execeute in an array or something like that).

Cron jobs and locking

Sometimes your cron jobs are so slow, the next call starts before your old finished (or you started it manually). In order to prevent your script from interfering with itself you want to use locking. Each job should use a different lockfile of course.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

LOCKFILE=~/.lock/my-scriptname.lock

(
  if ! flock -n 9; then
    echo "Couldn't lock '${LOCKFILE}', exit" >&2
    exit 1
  fi

  (
    # do stuff
    echo "long job!"
    sleep 100
    echo "done"
  ) 9>&-

) 9>>"${LOCKFILE}"

The inner subshell is used to hide the lockfile (in case you start daemons), 9>&- closes the file descriptor for the inner shell.

The outer subshell keeps the lockfile open, which is needed to hold the lock – the lock is released if either all fds are closed that held the lock, or if you call flock -u on one of them.

(I couldn’t find flock in posix specs – it works on linux and freebsd, but probably only on local filesystems. Shell doesn’t matter as flock is not a built-in.)

Security note: This does not create the LOCKFILE in a secure way, so don’t use world writable locations with this snippet.
I haven’t found a way yet to do this in a secure way; creating files with O_EXCL is not enough, if creation fails you also need to check the existing file to not be a symlink and be owned by you in one step (using results from the same lstat() call), and the standard “test” binary doesn’t provide this.

Also have a look at BashFAQ/045, it has some examples without flock (but they don’t recover if a previous run didn’t delete the lock file/directory).

Subshells

There is probably much more to say on this subject.. so just some small examples.

Subshells can be created with (commandlist), also asynchronous commands are executed in a subshell. Changes in subshells don’t affect the parent:

1

x=foo; (x=bar); echo $x

Commands in a pipe may be executed in a subshell (bash does this for all commands in a pipe, zsh for all but the last one), so this works in zsh but not in bash:

1

echo bar | read x; echo $x

Subshells inherit everything from the parent shell apart from the trap functions which are reset to their default behaviour.

Redirects

Note: POSIX only requires support for file descriptors 0 – 9 – for example dash and zsh are limited to single digit file descriptors.

This is only a short list – for details lookup some shell man pages (this is from the dash man page).

• [n]> file Redirect standard output (or n) to file; fails if “noclobber” option (-C) is active and the file exists and is a regular file.
• [n]>| file Same, but ignores noclobber.
• [n]>> file Append standard output (or n) to file.
• [n]< file Redirect standard input (or n) from file.
• [n]<&m Duplicate standard input (or n) from file descriptor m.
• [n]<&- Close standard input (or n). Not POSIX.
• [n]>&m Duplicate standard output (or n) to m.
• [n]>&- Close standard output (or n). Not POSIX.
• [n]<> file Open file for reading and writing on standard input (or n).

Reference: POSIX redirects

Special redirects

“Executing” redirects applies the redirects to the current (sub)shell.

Redirect stderr to a logfile:

1

exec 2>>/tmp/my-shell-script.log

Redirect stderr to stdout

1

exec 2>&1

Moving filedescriptors (bash only)

Move fd 6 to fd 5. Same as exec 6>&5 5>&-:

1

exec 5>&6-

Reverse pipe, process substitution (bash, zsh)

Above there was an example with subshells that fails in bash. Here is how to fix it:

1

read x < <(echo bar); echo $x

<(...) (and >(...)) is replaced by a filename to a pipe which is connected to the specified command.

Redirect multiple pipes to one process:

1

(cat <&3; read x; echo $x) 3< <(echo foo) < <(echo bar)

Safe iterating over file list from find (using nul-separated filenames; note the space between -d and ''):

1
2
3

while IFS= read -u3 -d '' -r file; do
  printf 'File found: "%s"\n' "$file"
done 3< <(find . -print0)

Be careful with using stdin in such loops – some programs read from stdin if it isn’t a tty device, assuming you piped them some commands. This is why I used fd 3; if you want to be extra careful you can close fd 3 for some commands/subshells in the loop.

Also check out Process Substitution