Wednesday, 6 June 2018

Using Cat << EOF in a shell script to restore binary files

Unix/Linux/Mac shell scripts support what is commonly called 'Cat << EOF' whereby an entire file can be included in a shell script and 'restored' to a separate file stored on the drive.

As should be obvious this derives from the fact that at the command line you can do a command such as -

cat /etc/hosts > newfile.txt

In this scenario we want to be able to use a shell script to create the desired file. You might think that this could be simply achieved as follows -

echo "This is the content of a file" > newfile.txt

and yes this to some extent is possible, however if your file is going to contain multiple lines - some of which might be blank lines and some might contain commands or special characters this soon becomes effectively impossible with a simple echo command. Therefore shell scripts can use the Cat << EOF feature instead. Here is a simple example -

cat <<- 'EOF' > newfile.txt
This is the content of a file
This is more content

Yet more content still

You should now be able to see where the reference to Cat << EOF comes from, however the official term is a 'heredoc', see

This approach is commonly used to include and generate a single text file as shown above. However what if you want to do something more complicated? What if you want to either do an entire directory, or nested set of directories/files or binary files? Is this even possible?

The answer fortunately is yes. To do this the easiest way I have found is as follows -

tar -cv nameofdirectory | openssl base64 -e

This uses the standard tar command to convert the specified directory (or binary files) and then pipes the result to openssl, openssl is then told to encode the input in to base64 format, base64 is an ASCII i.e. text encoded version of the binary data. In this case the result is then displayed to standard output in your terminal and can be copy/pasted in to your shell script in the Cat << EOF section.

Here therefore is an example shell script which would 'restore' a file

cd /where/you/want/to/restore
/usr/bin/openssl base64 -d << EOF | tar xf -

many lines deleted for the sake of readability


Whilst the command format is quite different it should be clear it is using the same openssl command to decode the base64 text and then sending it to the tar command to be 'restored'.

So this approach allows storing and restoring either a single or multiple binaries files or an entire hierarchy of directories and text or binary files. This approach can even cope with restoring an Apple disk image or an ISO image and after you restore it could then use another command to mount the image and copy or run something from that image.

No comments:

Post a Comment