Bash recipes

Parameter expansion

Convert wav files to mp3

Remove the file extension using parameter expansion.

for file in *.wav; do
> ffmpeg -i "$file" -acodec mp3 "${file%.*}.mp3"
> done

Loop files

Set basic ID3 tags for mp3 files

t=1; for file in *.mp3; do
> eyeD3 -a "Patrice O'Neal" -A 'Mr. P' -N 17 -n $t -G Comedy "$file"
> ((t++))
> done

Unpack multiple zip archives into one dir each based on zip file name

In this example I unpack files in my current workin dir into /media/nas/satashare/Comedy as a dir based on zip file name.

for file in *.zip; do
> mkdir -v "/media/nas/satashare/Comedy/${file%.*}"
> unzip "$file" -d "/media/nas/satashare/Comedy/${file%.*}/"
> done

Loop over only the latest copy of timestamped files

Let’s say you have a bunch of backup files named like this;

client1-20210725.tar.zstd
client1-20210719.tar.zstd
client2-20210725.tar.zstd
client2.20210719.tar.zstd

You want to automate unpacking these files but you only want to unpack the last of each client’s backup. As long as the timestamp format is correct you can combine an Array and a glob pattern to find the latest file.

while read -ru9 backup; do
    copies=($backup-202107*.zstd)
    zstd -d --stdout "${copies[-1]}" | ssh 'tar -xvf - -C /var/www'
done 9< <(grep -v '^#' clients.txt)
  • First of all the File Descriptor number 9 trick is explained further under Exhausting stdin on this page.
  • In this example we get a list of clients from a text file using grep, so we can comment out clients if we need to.
  • On line 2 we use a glob to get an array of all files matching $backup-202107*.zstd.
  • So then on line 3 we can use that array to get the latest file, based on the file name. So in this case the file names must be in order.

Random tricks

Convert from octal to decimal using printf

The terraform ignition provider takes file modes in decimal instead of octal for some reason, so since this is very unintuitive for me I had to convert them using printf.

printf '%d\n' 0755

Transfer many files over ssh

Use tar to pipe multiple files over the network into a sub-directory called my-server.

ssh my-server 'cd /var/log/ \
> sudo tar -cvf - maillog*' | tar -xvf - -C my-server/

Create a tar.gpg file when encrypting a folder

I sort of wish this was built-into tar just like zstd, xz and the like.

tar -cvf - my-files-directory | gpg2 --output my-files-directory.tar.gpg --encrypt -r 'My recipient key'

And to unpack (decrypt).

gpg2 --decrypt my-files-directory.tar.gpg | tar -tvf -

Pitfalls

Exhausting stdin

Stdout and stdin can be exhausted with other data, causing strange issues. Take this example;

while read -r line; do
    zstd -d --stdout "$line.tar.zstd" | ssh server 'tar -xvf - -C /var/www'
done < <(grep -v '^#' lines.txt)

The idea here is to go through a list of names from a file (lines.txt) and we use grep so that we can comment lines to skip them, then for each name unpack a file and send it over ssh to another server.

The problem here is that zstd is also sending data over stdin so this will only work for one iteration of the lines in the text file because after that stdin is exhausted by zstd and is missing the data from the sub-shell where grep was executed.

The correct way is to use another file descriptor for grep and while read to avoid exhausting it with zstd.

while read -ru9 line; do
    zstd -d --stdout "$line.tar.zstd" | ssh server 'tar -xvf - -C /var/www'
done 9< <(grep -v '^#' lines.txt)

This is not obvious but once you know it’s easy to remember that you can create file descriptors by prepending them to a redirection operator like this. And you can select it using while read -u.

Pre and post expansion

When incrementing a number with arithmetic expansion ((index++)) if index expands to a 1 it will actually return non-zero and exit any loop it was in.

The correct way is to increment pre-expansion like this; ((++index)).

See also