1. POSIX Specification

1.1 What is POSIX?

POSIX stands for "Portable Operating System Interface". It is a set of standards designed to maintain compatibility between various UNIX and UNIX-like operating systems. POSIX defines everything from system-level APIs and user facing interfaces. These standards make it easier for us to share programs.

1.2 Why use a POSIX shell?

In order to save ourselves time and effort, we should focus on writing portable code. In computing, "Portability" is the measure of how easy it is to transfer a program to a different environment. Since POSIX is a formal specification implemented by any operating system worth using, it is the Lowest Common Denominator. Writing POSIX programs means that we can write our program once, copy it to any system we might want to run it on, then run it with little to no extra effort required. POSIX means compatibility.

Although there are many shells to choose from, we should avoid any non-POSIX shells when scripting. Shells like bash , csh , zsh , and fish are not POSIX and therefore programs we write in them are not compatible with a pure POSIX system. Specifically, we must avoid bash and bashisms . bash is a popular shell but it is only present on systems infected with GNU and related GPL software. If we write our programs in bash they will not run on systems without GNU. We must target the lowest common denominator (POSIX) if we want our programs to be usable by everyone, everywhere.

If you want official documentation, see The POSIX sh specification

1.3 Anti-bash tips

  • Avoid long options. Ex: GNU systems support long options like "--verbose", POSIX systems only support short options like "-v".
  • There are no arrays, use IFS and plaintext.
  • On GNU systems, use man 1p to get the POSIX manual for a command
  • [[ option ]] is bash exclusive. Use [ option ] instead.
  • No local variables, use subshells

2. Shell Concepts

If you are completely new to the shell, see My guide on command line basics . This will help you get started. If you do not care to read something else, know that a UNIX system is a collection of small programs used via the shell. The user interacts with a UNIX system by the shell and can automate the system by creating shell scripts which are a series of commands placed into a script file. The shell is a fully featured programming language that makes it easy to create new programs on top of the existing UNIX utilities. All of this means that we can easily write simple programs without ever needing to learn a more complicated language like C. For most basic programs, shell scripts are more than enough.

2.1 What shell are we running?

Before we go any further, we should verify which shell we are running. Typically, we can just run echo $SHELL bu this will only tell us our default shell. In order to get the currently running shell , we should run echo $0 . The $0 variable contains the name of the currently running program. Below is an example of some various shells.

avery@nixphere:~ % echo $SHELL
/bin/tcsh
avery@nixphere:~ % bash
[avery@nixphere ~]$ echo $SHELL
/bin/tcsh
[avery@nixphere ~]$ echo $0
bash
[avery@nixphere ~]$ sh
$ echo $SHELL
/bin/tcsh
$ echo $0
sh
$ ^D
[avery@nixphere ~]$ 
exit
avery@nixphere:~ % 

Before we continue, we also need to verify that /bin/sh is actually a real program. If you have a GNU system, /bin/sh is actually a symlink to bash. If you are on a GNU system, you must start bash in POSIX mode with bash --posix . Modify your shebang accordingly. When you see #!/bin/sh , replace it with #!/bin/bash --posix . Additionally, when you see sh ./script.sh , replace it with bash --posix ./script.sh

This is a POSIX system

avery@nixphere:~ % file /bin/sh
/bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300139), FreeBSD-style, stripped
avery@nixphere:~ % 

And this is a GNU system

[avery@fedora ~]$ file /bin/sh
/bin/sh: symbolic link to bash
[avery@fedora ~]$ 

3. Basic Scripting

A shell script is quite literally a list of commands we want the shell to run for us. If we are clever, we can use this power to automate a series of actions. If we are sub-genius, we can use this power to write fully featured programs.

3.1 Shebang

To write a shell script, we simply create a plaintext file and open it in any editor. I will be using vim but you should replace this with your editor of choice. Typically, a shell script will end in the .sh extension. This is not required but it will help future you quickly determine what is and isn't a shell script.

avery@nixphere:~ % vim script.sh

UNIX Graybeards often refer to the first two characters of a shell script as a shebang . Some people call it a crunchbang but I call it a hashbang . These names are largely meaningless . . . but it's useful to remember that they stand for #! . Every shell script starts with a #! followed by whatever program you want to execute. Typically, the first like will be #!/bin/sh but if we wanted to create an awk script we could just as easily use #!/usr/local/bin/awk .

#!/bin/sh

3.2 Running Shell Scripts

First, let's write a simple shell script that tells us where we are at and lists our files. Notice that it's quite literally just a list of commands separated by newlines.

#!/bin/sh
pwd
ls

In order to run this script, we have two options. We can simply type sh script.sh

avery@nixphere:~ % sh script.sh
/home/avery
R		cr		public_html	src
bin		mbox		script.sh
avery@nixphere:~ % 

Or, if we want to run this program like a normal program, we can chmod u+x it then run it with ./script.sh

avery@nixphere:~ % chmod u+x ./script.sh
avery@nixphere:~ % ./script.sh 
/home/avery
R		cr		public_html	src
bin		mbox		script.sh
avery@nixphere:~ % 

3.3 Echo

echo is a command that allows us to print some text onto the screen. It's useful in that it can help us format our text and communicate progress ot the user. Let's modify script.sh from before.

#!/bin/sh
echo "you are here:"
pwd

echo "here are your files:"
ls
avery@nixphere:~ % ./script.sh 
you are here:
/home/avery
here are your files:
R		cr		public_html	src
bin		mbox		script.sh
avery@nixphere:~ % 

hmmm, we might be able to improve this output with command substitution . . .

3.4 Command Substitution With Echo

The shell allows us to use command substitution. Command substitution allows us to place a command within what we tell echo to say. This allows for greater formatting.

#!/bin/sh
echo "you are here: $(pwd)"

echo "here are your files: "

ls
avery@nixphere:~ % ./script.sh 
you are here: /home/avery
here are your files: 
R		cr		public_html	src
bin		mbox		script.sh
avery@nixphere:~ % 

3.5 Mixing Quotation

In shell scripts, quotation marks are important. When we surround multiple words in quotes, we create a string . Usually, the shell will split up our arguments, treating whitespace as a delimiter. To avoid chopping up our sentences, we use quotes. But there is a caveat: if we want to use quotation symbols inside our strings we must either mix quotation marks or escape them.

We can choose to begin and end our string with either a " or a ' . For now, this choice doesn't matter. Let's choose the " character. Notice how our doublequote is completely ignored

avery@nixphere:~ % echo "echo says "hello""
echo says hello
avery@nixphere:~ % echo "echo says 'hello'"
echo says 'hello'
avery@nixphere:~ % 

Alternative, we can use escape sequences. To escape the double quote, we can use the \ character. On some UNIX systems, echo cannot handle escape sequences. Instead, we can use the POSIX program printf . printf is somewhat strange in that escapes only work when our string is surrounded in single quotes. We must also supply a \n escape sequence if we want a newline after printf finishes.

avery@nixphere:~ % echo "echo says "hello""
echo says hello
avery@nixphere:~ % echo "echo says \"hello\""
Unmatched '"'.
avery@nixphere:~ % printf "echo says \"hello\""
Unmatched '"'.
avery@nixphere:~ % printf 'echo says \"hello\"'
echo says "hello"avery@nixphere:~ % 
avery@nixphere:~ % printf 'echo says \"hello\"\n'
echo says "hello"
avery@nixphere:~ % 

If you can, try to prefer to mix quotation marks instead of use escape sequences. Escape sequences are finicky and can be very difficult for humans to understand quickly.

3.6 Variables in Strings

In the shell, variables start with the $ character. We can place variables inside of strings

#!/bin/sh
echo "My home directory is '$HOME' and my username is '$USER'"

printf "here are my files: \n$(ls) \n"
avery@nixphere:~ % ./script.sh 
My home directory is '/home/avery' and my username is 'avery'
here are my files: 
R
bin
cr
mbox
public_html
script.sh
src 
avery@nixphere:~ %

3.7 Variables

The shell supports variables but there are some caveats. The name for a variable can contain letters, numbers, and underscore characters. Variable names cannot start with a space or a number. Also, spaces cannot be placed between the variable name, equals sign, and the value of the variable.

Good variables: 	| Bad variables:
------------------------+---------------
x=1			| x= 1
y=2			| y = 2
baz="foo bar"		| baz=foo bar
executables=$(ls /bin)  | exes =$(ls /bin)

let's add some variables to our shell script

#!/bin/sh
homedir="My home ditectory is"
username="and my username is"

echo "$homedir '$HOME' $username '$USER'"
avery@nixphere:~ % ./script.sh 
My home ditectory is '/home/avery' and my username is 'avery'
avery@nixphere:~ %

3.8 Comments

Comments allow us to annotate and explain out code. In shell, everything that comes after a # character is a comment. We can comment entire lines or only part of lines. Comments are also useful because they let us remove segments of code for testing and debugging purposes. Keep in mind, we can still escape the # character.

#!/bin/sh

# this is a comment

echo "hello world" # this line echoes hello world

# this function commented out because it's broken
#files=$(ls)
#for file in $files; do
#echo $flie
#done

echo \# done
avery@nixphere:~ % ./script.sh 
hello world
# done
avery@nixphere:~ %

3.9 Multiple commands on one line

The ; character can be used to separate our commands in a way that lets us write multiple commands on the same line. Unlike the && pipe, the ; separator will execute the following commands regardless of the exit code of the program before it.

avery@nixphere:~ % echo "hey mom" | figlet; echo "I'm on" | figlet; echo "UNIX" | figlet;

 _                                            
| |__   ___ _   _   _ __ ___   ___  _ __ ___  
| '_ \ / _ \ | | | | '_ ` _ \ / _ \| '_ ` _ \ 
| | | |  __/ |_| | | | | | | | (_) | | | | | |
|_| |_|\___|\__, | |_| |_| |_|\___/|_| |_| |_|
            |___/                             
 ___ _                         
|_ _( )_ __ ___     ___  _ __  
 | ||/| '_ ` _ \   / _ \| '_ \ 
 | |  | | | | | | | (_) | | | |
|___| |_| |_| |_|  \___/|_| |_|
	                                      
 _   _ _   _ _____  __
| | | | \ | |_ _\ \/ /
| | | |  \| || | \  / 
| |_| | |\  || | /  \ 
 \___/|_| \_|___/_/\_\
avery@nixphere:~ % 

3.10 Simple flow control

the || and && characters can be used to control flow. The || operator means "Or" and the && operator means "and"

The && operator will execute the command that follows it only if the exit code of the command before it is equal to 0.

The || operator will execute the command that follows it only if the exit code of the command before it is not equal to 0.

#!bin/sh
true && echo "this line is printed"
false || echo "this line also printed"
false && echo "this line is not printed"
true || echo "this line is not printed"
avery@nixphere:~ % sh ./control.sh 
this line is printed
this line also printed
avery@nixphere:~ %

3.11 Commands that span multiple lines

Just like how we can write multiple commands on one line, we can write one command on multiple lines using the \ character to escape the newlines. In this example, we are listing all users with a UID greater than 1000. In UNIX, all non-daemon users have a UID greater than 1000.

avery@nixphere:~ % cat /etc/passwd | sed s/:/\ /g | awk '$3>1000{print$1}'
nobody
avery
nuit
hadid
ra-hoor-khuit

is equivalent to

avery@nixphere:~ % cat /etc/passwd | \
? sed s/:/\ /g | \
? awk '$3>1000{print$1}'
nobody
nuit
hadid
ra-hoor-khuit
avery@nixphere:~ %

4. Environment

When we log in, our shell provides some basic environmental variables. They are called environmental variables because they provide information about (and configure) our environment.

4.1 Listing environmental variables

We can list them using either the env or set command.

avery@nixphere:~ % env
USER=avery
LOGNAME=avery
HOME=/home/avery
MAIL=/var/mail/avery
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/avery/bin
TERM=xterm-256color
BLOCKSIZE=K
MM_CHARSET=UTF-8
LANG=C.UTF-8
SHELL=/bin/tcsh
SSH_CLIENT=::1 58353 22
SSH_CONNECTION=::1 58353 ::1 22
SSH_TTY=/dev/pts/4
HOSTTYPE=FreeBSD
VENDOR=amd
OSTYPE=FreeBSD
MACHTYPE=x86_64
SHLVL=1
PWD=/home/avery
GROUP=avery
HOST=nixphere.org
REMOTEHOST=localhost
EDITOR=vim
PAGER=less
avery@nixphere:~ % 

the set command provides much more information

avery@nixphere:~ % set
_	env

addsuffix	
anyerror	
argv	()
autoexpand	
autolist	ambiguous
autorehash	
cdtohome	
csubstnonl	
cwd	/home/avery
dirstack	/home/avery
echo_style	bsd
edit	
euid	1002
euser	avery
filec	
gid	1003
group	avery
history	1000
home	/home/avery
killring	30
loginsh	
mail	/var/mail/avery
owd	
path	(/sbin /bin /usr/sbin /usr/bin /usr/local/sbin /usr/local/bin /home/avery/bin)
prompt	%{\033[1;32m%}%N@%m:%{\033[1;34m%}%~ %#%{\033[0m%} 
prompt2	%R? 
prompt3	CORRECT>%R (y|n|e|a)? 
promptchars	%#
savehist	(1000 merge)
shell	/bin/tcsh
shlvl	1
status	0
tcsh	6.21.00
term	xterm-256color
tty	pts/4
uid	1002
user	avery
version	tcsh 6.21.00 (Astron) 2019-05-08 (x86_64-amd-FreeBSD) options wide,nls,dl,al,kan,sm,rh,color,filec
avery@nixphere:~ % 
4.1.1 Environmental Variables Explained
$USER		your username
$LOGNAME	your login name
$HOME		/path/to/your/home/directory
$MAIL		/path/to/your/mailbox
$PATH		series of directories your shell searches for programs in
$TERM		terminal type
$LANG		your locale
$SHELL		your login shell
$OSTYPE		the UNIX variant on your computer
$MACHTYPE	your CPU's architecture
$PWD		the directory you are currently in
$GROUP		the primary group your user is in
$HOST		the hostname of your computer
$EDITOR		your default editor
$PAGER		your default pager

4.2 Combining variables with quotes and curly braces

We can combine our variables with strings using quotes and curly braces

avery@nixphere:~ % echo $OSTYPE" is cool"
FreeBSD is cool
avery@nixphere:~ % echo ${OSTYPE}\'s fast
FreeBSD's fast
avery@nixphere:~ %

4.3 Undefining variables

Sometimes we want to remove a variable from our program. There are a couple ways to do it.

The first way keeps the variable in our program but removes the value we've assigned to it.

variable=

The second way actually removes the variable from our program entirely

unset variable

Here is a demonstration. We are using the set command to list all variables, then piping it's output through grep to check if there is a match. The grep command will return 1 if there were no matches.

#!/bin/sh
variable="hello world"

set | grep variable
echo "Exit code: $?"

variable=
set | grep variable
echo "Exit code: $?"

unset variable
set | grep variable
echo "Exit code: $?"
avery@nixphere:~ % sh ./vars.sh 
variable='hello world'
Exit code: 0
variable=''
Exit code: 0
Exit code: 1
avery@nixphere:~ % 

5. Command line arguments

We can pass arguments to our shell script just like with other commands. These are indexed, in order, as variables. $1 represents the first argument, $2 represents the second argument, $3 represents the third, etc. $0 is special in that it contains the name of the program.

5.2 rename.sh

Let's write a script called rename.sh that we can use to rename files.

#!/bin/sh
mv -i $1 $2
avery@nixphere:~ % touch foo
avery@nixphere:~ % ./rename.sh foo bar
avery@nixphere:~ % ls
R           bin         mbox        recycle.sh  script.sh
bar         cr          public_html rename.sh   src
avery@nixphere:~ % touch baz
avery@nixphere:~ % ./rename.sh baz bar
overwrite bar? (y/n [n]) y
avery@nixphere:~ % ls
R           bin         mbox        recycle.sh  script.sh
bar         cr          public_html rename.sh   src
avery@nixphere:~ % 

5.2 recycle.sh

Let's write a script called recycle.sh that we can use to place our files into a recycling bin. This script contains some concepts we have not covered yet.

$? is a variable that holds the exit code of the last program. Every well behaved program in UNIX has an exit code. An exit code other than 0 indicates an error.

We are also using an if statement to compare the exit code of the last command (the move command) to zero. The -eq flag stands for "equals". If $? == 0 , then we print the success message. We'll learn more about this later.

#!/bin/sh
recyclebin="$HOME/.recyclebin"

# uncomment if $HOME/.recyclebin does not exist
# mkdir $recyclebin

# move file to recycle bin
mv $1 $recyclebin/$1

# if the file was sucessfully moved, print a success message
if [ $? -eq 0 ]; then
	echo $0 has sucessfully recycled $1
fi
avery@nixphere:~ % ./recycle.sh foo
./recycle.sh has sucessfully recycled foo
avery@nixphere:~ % ./recycle.sh bar
mv: rename bar to /home/avery/.recyclebin/bar: No such file or directory
avery@nixphere:~ %

5.3 Number of parameters

The $# variable represents the number of arguments we passed to our program. We can use this to loop through all of the supplied arguments. The $@ variable represents all command line arguments.

In some shells, a plain old $@ gets ignored so for maximum portability we should use ${1+"$@"} instead. This format basically translates to "if $1 is defined, do something. If $i is not defined, do nothing."

We are also using a for loop here. For loops iterate until some condition is met. We'll learn more about this later.

#!/bin/sh
echo "total number of arguments: $#"

for i in ${1+"$@"}; do
	echo $i
done
avery@nixphere:~ % ./counter.sh foo bar baz
total number of arguments: 3
foo
bar
baz
avery@nixphere:~ % 

6. Loops and Logic

In programming, a loop statement can be used to repeat a command n times. This means we can write less code that does more work. When we add logic statements, we can make decisions. Combining loops and logic allows us to create very powerful tools.

6.0.1 Conditional expressions

Comparison operators in the shell might be confusing at first. Intead of using mathematical operators, we use a - character followed by whaver condition we are testing for. Mathetimacal operators are bashisms and should be avoided to ensure maximum portability.

These operators are used to compare two variables or values. We can use these expressoins to make decisions in our programs.

mathematical expression shell equivalent what it does Does the mathematical expression works in POSIX shell?
== -eq tests if equal YES
!= -nq tests if not equal YES
< -lt tests if less than NO
> -gt tests if greater than NO
<= -le tests if less than or equal to NO
>= -ge tests if greater than or equal to NO

6.1 If

An if statement allows us to make decisions based on some condition. It follows the general structure of if expression is equal to value then do action . Notice how each if statement ends with "fi". "fi" is "if" but backwards. Notice how the statement do continue after one of the checks is true.

#!/bin/sh
a=$1
b=$2

if [ $a -eq $b ]
	then
	echo "they are equal"
fi

if [ $a -ne $b ]
	then
	echo "they are not equal"
fi
avery@nixphere:~ % sh ./equality.sh 10 20
they are not equal
avery@nixphere:~ % sh ./equality.sh 10 10
they are equal
avery@nixphere:~ % 
6.1.1 If-Elseif-Else

What if we wanted to write a more concise if statement? We can use the "elif" statement. "elif" is short for "else if" and is used to check for a secondary condition if the first test is false. Notice how the statement does not continue after one of the checks is true. The "else" statement is used as a backup of sorts. If none of the statements before the "else" are true, then whatever is inside of the else block is done.

Also notice how we are placing if and then on the same line. We are separating them with the ; character.

#!/bin/sh
a=$1
b=$2

if [ $a -eq $b ]; then
	echo "they are equal"
elif [ $a -ne $b ]; then
	echo "they are not equal"
elif [ $a -eq 19 ] && [ $b -eq 70 ]; then # check if $a is 19 AND $b is 70
	echo "$a​$b, an easter egg for the UNIX epoch"
else 
	echo "some error occured"
fi
avery@nixphere:~ % sh equality.sh 19 70
they are not equal
avery@nixphere:~ % 

Let's modify our shell script so that we can see the easter egg. We need to move the conditional with the highest priority to the top of our if-else-chain. Let's also add an exit code to our final else statement to indicate that there was some error in the program if it happens to go through all our checks without matching one.

#!/bin/sh
a=$1
b=$2

if [ $a -eq 19 ] && [ $b -eq 70 ]; then 	# check if $a is 19 AND $b is 70
	echo "$a​$b, an easter egg for the UNIX epoch"
elif [ $a -eq $b ]; then
	echo "they are equal"
elif [ $a -ne $b ]; then
	echo "they are not equal"
else 
	echo "some error occured"
	exit 1				# exit with an error
fi
avery@nixphere:~ % sh ./equality.sh 19 70
1970, an easter egg for the UNIX epoch
avery@nixphere:~ % 

6.2 Case

A case statement (sometimes called a switch statement) is similar to to an if-else statement but with more options. Notice how we end the case statement with 'esac'. 'esac' is just 'case' spelled backwards. Every block ends with the ;; character. These mean "Hey we are done doing $stuff, go to the next check".

#!/bin/sh
printf "Answer Yes or No: "
read userinput # read command gets input from the user
case $userinput in
        yes | YES ) 	# if $userinput matches yes or YES
                echo "You answered yes"
                ;;
        no | NO )	# if $userinput matches no or NO
                echo "You answered no"
                ;;
esac
avery@nixphere:~ % sh cases.sh 
Answer Yes or No: yes
You answered yes
avery@nixphere:~ % sh cases.sh
Answer Yes or No: YES
You answered yes
avery@nixphere:~ % sh cases.sh
Answer Yes or No: no
You answered no
avery@nixphere:~ % sh cases.sh
Answer Yes or No: No
avery@nixphere:~ % 

Hmmm, looks like some of our arguments got ignored. Since we are only checking for all caps or all lowercase, a mixed case word is ignored. Let's use some regex to match more generously.

Instead of an "else" catching anything that slips through the cracks, a case statement uses "default". Just like in regex, the * character matches everything. Let's also add a "default" block to catch any user input that doesn't get matched.

#!/bin/sh
printf "Answer Yes or No: "
read userinput # read command gets input from the user
case $userinput in
        [yY][eE][sS] )  # check for mixed case y-e-s
                echo "You answered yes"
                ;;
        [nN][oO] )      # check for mixed case n-o
                echo "You answered no"
                ;;
        *)              # what to do if we didn't get a match
                echo "I couldn't understand '$userinput'"
                exit 1  # exit with an error
                ;;
esac
avery@nixphere:~ % sh cases.sh 
Answer Yes or No: yEs
You answered yes
avery@nixphere:~ % sh cases.sh 
Answer Yes or No: No
You answered no
avery@nixphere:~ % sh cases.sh 
Answer Yes or No: YeS
You answered yes
avery@nixphere:~ % sh cases.sh 
Answer Yes or No: I don't want to answer
I couldn't understand 'I don't want to answer'
avery@nixphere:~ %

6.3 Break/Continue

The "for", "while", and "until" loops execute the same lines of code more than once. The "break" command will immediately exit the loop. The "continue" command will immediately restart the loop from the top. Keep this in mind as we go forward.

6.4 While

A "while" loop runs while some condition is true. When the condition is no longer true, the loop stops. A while loop is useful for processing user input or for executing a section of code multiple times until the desired number of iterations is achieved.

#!/bin/sh
i=0

# loop while $i is less than or equal to 10
while [ $i -le 10 ]; do
        echo "Loop #$i"
        i=$(( i + 1 ))	# increment $i by 1 every loop
done
avery@nixphere:~ % sh while.sh
Loop #0
Loop #1
Loop #2
Loop #3
Loop #4
Loop #5
Loop #6
Loop #7
Loop #8
Loop #9
Loop #10
avery@nixphere:~ %

We can also use while loops to create infinite loops.

#!/bin/sh

while [ true ]; do
        echo "I can't stop aaaaAAA"
done
avery@nixphere:~ % sh while.sh 
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
^C
avery@nixphere:~ %
6.4.1 Color generator with while loops

Let's use a while loop to generate all 256 colors supported by a 256-color terminal emulator. We can use ANSI escape sequences to make the colors

#!/bin/sh
i=0
while [ $i -le 255 ]; do
        printf "\033[48:5:${i}m  \033[0m"
        i=$(( i + 1 ))
done

printf "\n"
avery@nixphere:~ % ./256test.sh 
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
avery@nixphere:~ %

6.5 Until

An until loop is very similar to a while loop. An until loop will continue looping until some condition is met. Some people consider until loops to be more danger than while loops but they're almost identical.

#!/bin/sh
i=0

# loop until $i is equal to 10
until [ $i -eq 10 ]; do
        echo "Loop #$i"
        i=$(( i + 1 ))	# increment $i by 1 every loop
done
avery@nixphere:~ % sh while.sh
Loop #0
Loop #1
Loop #2
Loop #3
Loop #4
Loop #5
Loop #6
Loop #7
Loop #8
Loop #9
avery@nixphere:~ %
6.5.1 Color generator with until loops

Let's use an until loop to generate all 256 colors supported by a 256-color terminal emulator.

#!/bin/sh
i=0
until [ $i -eq 256 ]; do
        printf "\033[48:5:${i}m  \033[0m"
        i=$(( i + 1 ))
done

printf "\n"
avery@nixphere:~ % ./256test.sh 
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
avery@nixphere:~ %

6.6 For

A for loop operates on a list of items. It's useful for iterating over a series of files or a string delimited by spaces. Since there are no arrays in POSIX shell, we can use a string instead. A for loop will do an action for every item in the list we provide.

#!/bin/sh

# iterating over a list
for item in 1 2 3 4 5; do 
        echo "Loop #$item"
done

# no arrays, only strings :)
string="one two three four five"

#iterating over a list of items stored in a variable
for item in $string; do
        echo "Number $item"
done
avery@nixphere:~ % sh for.sh 
Loop #1
Loop #2
Loop #3
Loop #4
Loop #5
Number one
Number two
Number three
Number four
Number five
avery@nixphere:~ %
6.6.1 Gallery program with for loops

This script is . . . not entirely portable. It requires a program called 'viu'. You can probably use your package manager to install it. If not, you can build it from the viu repo

#!/bin/sh

while true; do
        # make pictures variable empty
        pictures=
        # fill pictures variable with a list of files, randomly sorted
        pictures=$(ls Pictures/ | sort -R)

        for item in $pictures; do       # loop through files
                clear                   # clear screen
                viu Pictures/$item      # display image with viu
                sleep 2                 # wait 2 seconds
        done

done

Here is a demo video:

But I think we can improve this gallery tool using other skills we've learned so far.

#!/bin/sh
# the path to our images is supplied the command line 
path=$1 

# use test command, -z flag tests if the string is empty
# if there are no arguments (ie empty string), then exit with an error code
if test -z $path ; then 
        echo "you must supply a path to your images"
        exit 1
fi

echo "press enter to go to the next image"
echo "press q then enter to quit"
echo 
echo "press enter to continue"

# wait for enter key
read keypress

# fill pictures variable with a list of files 
pictures=$(ls -d ${PWD}/${path}/*)

for item in $pictures; do       # loop through files
        clear                   # clear screen
        viu $item               # display image with viu
        printf "\033[1mImage: \033[0m$item\n"   # display image name

        read keypress           # wait for keypress
        case $keypress in
                q)              # if q then enter, exit program
                        break
                        ;;
                *)              # any key or enter, continue
                        continue
                        ;;
        esac
done

Here is a demo of our improved program:

7. Math, Tests, and other Expressions

7.1 Basic math, incriminating, and decrementing

We don't need to remember fancy programs for very basic mathematical operations because we can call the eval command with a fancy trick that looks a lot like command substitution. Let's build a loop that calculates the Fibonacci sequence to demonstrate.

Here we are doing some math. We need four variables: a counter, the current number, the last number, and the last last number. We then run a test that will safely exit the program if the user does not specify a number of operations. After that, we run a loop. "Continue looping while the number of iterations is less than the number specified by the user". Inside this loop, we do the math required to generate a Fibonacci sequence. At the very end, we print a newline for clean output.

the spaces within the $(( [ math here ] )) are significant! It might work without spaces for you but the eval program on someone else's system might not work without the spaces!

#!/bin/sh
i=0     # iteration counter
a=1     # start at 1
b=0     # remember last number
c=0     # remember last last number

# safety check to prevent messy errors
if test -z $1; then
        echo "please specify a number of iterations"
        exit 1
fi

# while the number of iterations is less than or
# or equal to the number specified in the first argument, $1
while [ $i -le $1 ]; do
        printf "$a "            # print value
        c=$b                    # last last value = last value
        b=$a                    # last value = current 
        a=$(( $c + $b ))        # current = last + last last

        i=$(( $i + 1 ))         # iterate counter by 1
done

printf "\n" # for formatting 
avery@nixphere:~ % ./fib.sh 
please specify a number of iterations
avery@nixphere:~ % ./fib.sh 10
1 1 2 3 5 8 13 21 34 55 89 
avery@nixphere:~ % ./fib.sh 100
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765 10946 17711 28657 46368 75025 121393 196418 317811 514229 832040 1346269 2178309 3524578 5702887 9227465 14930352 24157817 39088169 63245986 102334155 165580141 267914296 433494437 701408733 1134903170 1836311903 2971215073 4807526976 7778742049 12586269025 20365011074 32951280099 53316291173 86267571272 139583862445 225851433717 365435296162 591286729879 956722026041 1548008755920 2504730781961 4052739537881 6557470319842 10610209857723 17167680177565 27777890035288 44945570212853 72723460248141 117669030460994 190392490709135 308061521170129 498454011879264 806515533049393 1304969544928657 2111485077978050 3416454622906707 5527939700884757 8944394323791464 14472334024676221 23416728348467685 37889062373143906 61305790721611591 99194853094755497 160500643816367088 259695496911122585 420196140727489673 679891637638612258 1100087778366101931 1779979416004714189 2880067194370816120 4660046610375530309 7540113804746346429 printf: Illegal option -6
1293530146158671551 printf: Illegal option -4
printf: Illegal option -3
printf: Illegal option -8
6174643828739884737 printf: Illegal option -2
3736710778780434371 1298777728820984005 
avery@nixphere:~ %

Oh no! it looks like the number got too large and caused a buffer overflow in expr . Let's try to re-write this program so that it can calculate the first 100 numbers in the Fibonacci sequence.

7.2 bc

bc is a slightly more robust and feature complete and extensible math program(ming language). We can remember bc by thinking that it stands for "basic calculator". By default, bc is very similar to expr but it supports larger numbers. We can do square roots, absolute values, arithmetic, etc.

Let's re-write our program to use bc Instead of using the method from the previous section, we are using command substitution to do our math.

#!/bin/sh
i=0     # iteration counter
a=1     # start at 1
b=0     # remember last number
c=0     # remember last last number

# safety check to prevent messy errors
if test -z $1; then
        echo "please specify a number of iterations"
        exit 1
fi

# while the number of iterations is less than or
# or equal to the number specified in the first argument, $1
while [ $i -le $1 ]; do
        printf "$a "            # print value
        c=$b                    # last last value = last value
        b=$a                    # last value = current 
        a=$( echo $c + $b  | bc )       # current = last + last last

        i=$( echo $i + 1 | bc )         # iterate counter by 1
done

printf "\n" # for formatting 
avery@nixphere:~ % ./fib-bc.sh 
please specify a number of iterations
avery@nixphere:~ % ./fib-bc.sh 10
1 1 2 3 5 8 13 21 34 55 89 
avery@nixphere:~ % ./fib-bc.sh 100
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765 10946 17711 28657 46368 75025 121393 196418 317811 514229 832040 1346269 2178309 3524578 5702887 9227465 14930352 24157817 39088169 63245986 102334155 165580141 267914296 433494437 701408733 1134903170 1836311903 2971215073 4807526976 7778742049 12586269025 20365011074 32951280099 53316291173 86267571272 139583862445 225851433717 365435296162 591286729879 956722026041 1548008755920 2504730781961 4052739537881 6557470319842 10610209857723 17167680177565 27777890035288 44945570212853 72723460248141 117669030460994 190392490709135 308061521170129 498454011879264 806515533049393 1304969544928657 2111485077978050 3416454622906707 5527939700884757 8944394323791464 14472334024676221 23416728348467685 37889062373143906 61305790721611591 99194853094755497 160500643816367088 259695496911122585 420196140727489673 679891637638612258 1100087778366101931 1779979416004714189 2880067194370816120 4660046610375530309 7540113804746346429 12200160415121876738 19740274219868223167 31940434634990099905 51680708854858323072 83621143489848422977 135301852344706746049 218922995834555169026 354224848179261915075 573147844013817084101 
avery@nixphere:~ %

By default, bc will automatically floor (ie round down to the nearest whole number). To get decimal precision, we can include the scale indicator in the string we echo to bc

avery@nixphere:~ % echo "1/3" | bc
0
avery@nixphere:~ % echo "scale=2; 1/3" | bc
.33
avery@nixphere:~ % echo "scale=4; 1/3" | bc
.3333
avery@nixphere:~ % echo "scale=1; .3 * 7" | bc
2.1
avery@nixphere:~ % 

If we run bc -l , we will get support for trig, exponentials, logs, etc. If you need more than the basic arithmetic shown above, you should read man bc(1) . I cannot demonstrate it all here.

7.3 test

The test command is another useful tool for shell scripting. It can be used to easily test for a wide variety of things we might need in a script. If the test is true, test will return 0. If the test is false, test will return a 1.

I have included some of the most useful tests but you really should read the man page. It's much more useful than this.

test what it does
-e file True if file exists (regardless of type).
-n string True if the length of string is nonzero.
-s file True if file exists and has a size greater than zero.
-z string True if the length of string is zero.
file1 -ef file2 True if file1 and file2 exist and refer to the same file.
string True if string is not the null string.
s1 = s2 True if the strings s1 and s2 are identical.
s1 != s2 True if the strings s1 and s2 are not identical.

The test command also supports multi-condition statements. The -a operator has higher precedence than the -o operator.

! expression True if expression is false.
expression1 -a expression2 True if both expression1 and expression2 are true.
expression1 -o expression2 True if either expression1 or expression2 are true.
( expression ) True if expression is true.

7.4 expr

The expr command can be used for expression evaluation. If we pass a mathematical operation, expr will return the result. If we pass a any other type of operator, expr will return 0 for false, 1 for true, and 2 for a program error.

7.4.1 Arithmetic

The mathematical operators are very simple but there is one caveat: you must place spaces around all operators and escape the following characters if you use them in your operation: *&|><()/ .

avery@nixphere:~ % expr 1 + 1
2
avery@nixphere:~ % expr 2 - 1
1
avery@nixphere:~ % expr 2 * 2
expr: syntax error
avery@nixphere:~ % expr 2 \* 2
4
avery@nixphere:~ % expr 4 \/ 2
2
avery@nixphere:~ % expr 7 % 5
2
7.4.1 Relational

The relational operators should be familliar. We've used them before. As with many other programs, expr will return 0 for a true statement and 1 for a false statement.

#!/bin/sh
# list of variables
a=$( expr 2 "=" 1  )
b=$( expr 2 ">" 1  )
c=$( expr 2 ">=" 1  )
d=$( expr 2 "<" 1  )
e=$( expr 2 "<=" 1  )
f=$( expr 2 "!=" 1  )

# print our variable assignment 
cat expr.sh | grep " expr" | grep -v cat | grep =


# set delim character to .
IFS="."

# make an array of the return codes form expr
# echo will print all vars in order, making
# a list separated by the delim character 
# that we can iterate over
arr=$( echo $a.$b.$c.$d.$e.$f)



# loop through the array
for i in $arr; do
        if [ $i -eq 1 ]; then
                printf "$i true"
        elif [ $i -eq 0 ]; then
                printf "$i false"
        elif [ $i -eq 2 ]; then
                printf "$i ERROR"
        fi

printf "\n"
done
avery@nixphere:~ % sh expr.sh
a=$( expr 2 = 1  )
b=$( expr 2 ">" 1  )
c=$( expr 2 ">=" 1  )
d=$( expr 2 "<" 1  )
e=$( expr 2 "<=" 1  )
f=$( expr 2 "!=" 1  )
0 false
1 true
1 true
0 false
0 false
1 true
avery@nixphere:~ % 
7.4.3 Boolean

The boolean operator works on both strings and integers. Null strings and variables are false. Non-null strings and vars are true. 1 = true, 0 = false.

avery@nixphere:~ % expr 1 \& 1
1
avery@nixphere:~ % expr 1 \& 0
0
avery@nixphere:~ % expr 1 \| 1
1
avery@nixphere:~ % expr 1 \| 0
1
avery@nixphere:~ % expr 0 \| 0
0
avery@nixphere:~ % 
7.4.4 String

The string operator, : can be used to compare strings. Any number greater than 0 is true, any number equal to zero is false.

avery@nixphere:~ % expr foo : foo
3
avery@nixphere:~ % expr foo : bar
0
avery@nixphere:~ % expr food : "foo*"
3
avery@nixphere:~ % expr baz : ".*"
3
avery@nixphere:~ %

8. Functions

9. Multithreading

9.1 Background Jobs

In the terminal, we can start a process in the background by placing a & character at the end of the line. When using this method, it's important to use the wait command before continuing the script. wait will wait untill all background jobs are completed before continuing.

Let's build a multithreaded image converter using ImageMagick. This program isn't part of the POSIX spec so you'll need to install it via your package manager or download ImageMagick here . We'll be using ImageMagick to convert all of our images to .jpgs.

#!/bin/sh
# get array of all pictures, excluding subdirectories
files=$(ls -p ~/Pictures | grep -v \/)

cd ~/Pictures

# check if converted dir exists, if not create it
if ! test -e converted; then
        mkdir converted
fi


# counter for impatient users
echo "converting images, please wait"
j=1 

# iterate through files
for i in $files; do

        # using sed to remove the file extension
        filename=$(echo $i | sed 's/\.[^\.]*$//')

        # convert file to converted/file.png
        convert $i converted/${filename}.png &

        # counter for impatient users
        echo "starting job $j"
        j=$(( j + 1 ))
done

# wait until all jobs are finished
wait

# echo then exit
echo "done"
avery@nixphere:~ % sh ./multithread.sh
converting images, please wait
starting job 1
starting job 2
starting job 3
starting job 4
starting job 5
starting job 6
starting job 7
starting job 8
starting job 9
starting job 10
starting job 11
done
avery@nixphere:~ % ls Pictures/
1616854535700.jpg 1621241813304.jpg antinatalist.jpg  machine.jpg
1618463408483.jpg 1623754061681.png argch1.png        scrot.png
1619699442668.jpg 1625650766474.png converted         sun.jpg
avery@nixphere:~ % ls Pictures/converted/
1616854535700.png 1621241813304.png antinatalist.png  scrot.png
1618463408483.png 1623754061681.png argch1.png        sun.png
1619699442668.png 1625650766474.png machine.png
avery@nixphere:~ %

This seems to work quite well, but there is one issue: we can lose efficiecy at some point. Starting more jobs than we have resources might create a bottleneck on our system. Be it limited CPUs, limited memory, disk, or network - we can actually slow down our system overall.

9.2 xargs

The xargs command solves the problem from above. It does most of the hard work for us. Let's re-write our program with xargs so that we can do multithreading without grinding the system to a halt.

#!/bin/sh
# get array of all pictures, excluding subdirectories
maxjobs=$1
files=$(ls -p ~/Pictures | grep -v \/)

cd ~/Pictures

# check if converted dir exists, if not create it
if ! test -e xargs; then
        mkdir xargs
fi

echo $files | sed 's/\ /\n/g' | xargs -P $1 -I '{}' convert '{}' xargs/'{}'.png

# echo then exit
echo "done"

10. Cron Jobs

Not finished yet but I'm working on it! Check back soon!

Written by Avery. This work is licensed under a CC BY-SA 4.0 License