1. History and Design

1.1 From Wikipedia:

A Unix shell is a command-line interpreter or shell that provides a command line user interface for Unix-like operating systems. The shell is both an interactive command language and a scripting language, and is used by the operating system to control the execution of the system using shell scripts.

Users typically interact with a Unix shell using a terminal emulator; however, direct operation via serial hardware connections or Secure Shell are common for server systems. All Unix shells provide filename wildcarding, piping, here documents, command substitution, variables and control structures for condition-testing and iteration.

The most generic sense of the term shell means any program that users employ to type commands. A shell hides the details of the underlying operating system and manages the technical details of the operating system kernel interface, which is the lowest-level, or "inner-most" component of most operating systems.

- Unix Shell - Wikipedia

Before we begin, we should go over some history and basic systems design. UNIX systems originate from Bell Labs in the 1970s. Computers in the 1970s were much less powerful than they were today. Programs needed to be small due to the constrained nature of the systems at the time. Here is a PDP7 - a computer similar to what UNIX was first designed for. It's a big mainframe system. If you look closely, you'll see that there is no screen! Only a panel of switches, an oscilloscope, a few tape drives, and something that looks like a typewriter. In order to use these machines, the operator would sit at the teletype terminal and enter commands for the system to run. As the system finished running commands, the output would be printed on the same terminal. Since there was no graphical display, programs needed to be simple enough to run with minimal user interference.

Eventually, as these systems became more powerful, they were able to support multiple users at once. As technology progressed, users moved from teletypes to dumb terminals. A dumb terminal was an upgrade because it supported primitive character graphics and even early windowing systems. They were called 'dumb' because they actually only contained just enough hardware and processing power to connect to a remote terminal server (think PDP11). All computing still happened on the mainframe.

Keeping these constraints in mind, we can proceed to look at the design of a UNIX system. A UNIX system is very simple in that it is composed of simple logical components:

Hardware - The physical silicon crunching numbers.

Kernel - The kernel is responsible for managing resources, system processes, and communications with the hardware. The kernel is largely immutable and invisible in that the average user rarely (if ever) needs to manually communicate with the hardware.

Shell - The shell sits on top of the kernel and is how a user interfaces with the system. The shell is a friendly and simple environment designed to get work done. It take commands from the users. The shell knows where all the programs are at starts them upon the user's command.

Utilities - The utilities simultaneously sit on top of and alongside the shell. Although it is possible to run utilities without a shell, it is not an easy or pleasant experience. These programs are usually very small an designed to be strung together to create larger, more useful tools.

1.2 Back to the Shell

The Shell is a text-only interface to a UNIX system. A common misconception is that the shell is hard to learn but consider the following: the shell reads from human language and does an action based on the words. A modern day voice assistant is a similar concept. You tell the computer to do action with modifiers on a Subject and a direct object. If you can use a voice assistant, you can use a terminal.

Another terminal misconception is that it's old, inefficient, and useless. This couldn't be more incorrect. Modern UNIX systems are actually very comfortable to use. Although the shell and it's utilities retain backwards compatibility, many modern terminal programs actually have so many features that they won't work on a teletype or dumb terminal! As you learn the terminal, you will see efficiency gains as you climb the learning curve. It's always faster to type out exactly what you want the computer to do than it is to click through menu after menu, dragging files between programs, manually doing each and every step that could be automated. As for usefulness, everything but your laptop subsidized by microsoft runs some variant of UNIX. Every webserver, every smartphone, every macbook, every playstation, every internet connected smart toaster. It seems like everyone who begins learning UNIX and the shell also increase their skills in general logic, programming, process structuring, and computing in general.

1.3 Naming Conventions

Many UNIX terms are flexible and everyone calls them differently. This can cause confusion for new users. The Shell is also sometimes called the Terminal, the Command Line, the Prompt, Bash, or the Command Interpreter. All of these terms are interchangeable and you can pick whichever you like. Lately I've been calling it 'shell' because it implies UNIX as opposed 'command prompt' which can imply DOS to a room of paid microsoft shills.

Another confusing thing can be the difference between Linux and UNIX. UNIX is a specification based on the POSIX standards. Linux, *BSD, illumos, etc are simply different implementations of UNIX. For the purposes of this article, Linux and UNIX are interchangeable terms unless otherwise specified. This statement is factually incorrect . . . but this doesn't really matter if you are just starting out.

2. Basic Navigation

Here we will actually begin playing with the command line. Be sure to open your terminal and follow along.

2.1 Case Sensitivity

On UNIX, everything is 'case sensitive'. This means that uppercase and lowercase matters! make sure to follow the examples exactly. pwd will work but PWD will not work. Similarly, a file named foo.txt is different from FOO.txt. This shouldn't be a huge issue unless you have CP/M Stockholm syndrome.

2.2. Understanding the prompt

Since the shell is highly customizable, your prompt will most certainly look different. On non-brutalist systems and on Nixphere (if you chose Avery's custom configs) your prompt will look something like this:

 avery@nixphere:~ % 

Let's examine this. The first thing you see is your username, then an @, then the hostname of the machine you are logged into. After the user@host, we have a ~ symbol. This will change to to the directory you are currently in. Finally, there is a % symbol. In many shells, you might only see a $, %, >, or # symbol without color. All this symbol means is that the shell is listening for commands, ready to take your input.

The terms 'directory' and 'folder' are interchangeable. Similarly, 'subdirectory' is interchangeable with 'subfolder'.

2.3 Where are we at?

The pwd command stands for 'Print Working Directory'. As you'll soon see, many UNIX programs have short, mnemonic names or abbreviations. pwd tells us what directory (folder) you are currently in. This is useful for when you forget where you are.

avery@nixphere:~ % pwd
/home/avery
avery@nixphere:~ %

2.4 What's here?

The ls command stands for 'List'. We can append flags after the name of the command. These flags act as modifiers to the behavior of the program. The -a flag stands for 'All' and lists hidden files. In UNIX, naming a file with a period in front will hide it. The -l flag stands for 'Long' and will show us extra information about our files. We can combine flags too!

avery@nixphere:~ % ls
R           mbox        public_html
avery@nixphere:~ % ls -l
total 6
drwxr-xr-x  3 avery  avery     3 Jul 31 00:32 R
-rw-------  1 avery  avery  1426 Jul  3 20:52 mbox
lrwxr-xr-x  1 avery  avery    35 Jun 30 01:18 public_html -> /usr/local/www/apache24/data/~avery
avery@nixphere:~ % ls -la
total 66
drwxr-xr-x  7 avery  avery     20 Aug  1 00:17 .
drwxr-xr-x  5 root   wheel      5 Jul 31 16:32 ..
-rw-r--r--  1 avery  avery    420 Jul 31 00:59 .Rhistory
drwxr-xr-x  4 avery  avery      4 Jul 31 00:45 .config
-rw-r--r--  1 avery  avery   1248 Jun 30 01:00 .cshrc
-rw-------  1 avery  avery  15728 Jul 31 21:24 .history
-rw-------  1 avery  avery     73 Jul 13 01:58 .lesshst
drwxr-xr-x  3 avery  avery      3 Jul 31 00:32 .local
-rw-r--r--  1 avery  avery    323 Jun 30 01:00 .login
-rw-r--r--  1 avery  avery     91 Jun 30 01:00 .login_conf
-rw-------  1 avery  avery    301 Jun 30 01:00 .mail_aliases
-rw-r--r--  1 avery  avery    267 Jun 30 01:00 .mailrc
-rw-r--r--  1 avery  avery    978 Jun 30 01:00 .profile
-rw-r--r--  1 avery  avery    695 Jun 30 01:00 .shrc
drwxr-xr-x  2 avery  avery      4 Jul  3 16:13 .ssh
drwxr-xr-x  2 avery  avery      3 Jul  3 20:40 .vim
-rw-------  1 avery  avery  23628 Jul 27 00:37 .viminfo
drwxr-xr-x  3 avery  avery      3 Jul 31 00:32 R
-rw-------  1 avery  avery   1426 Jul  3 20:52 mbox
lrwxr-xr-x  1 avery  avery     35 Jun 30 01:18 public_html -> /usr/local/www/apache24/data/~avery
avery@nixphere:~ %

The ls command also takes arguments. Let's try it out.

avery@nixphere:~ % ls /bin
[               cp              domainname      freebsd-version link            pax             pwd             rmdir           sync
cat             csh             echo            getfacl         ln              pgrep           realpath        setfacl         tcsh
chflags         date            ed              hostname        ls              pkill           red             sh              test
chio            dd              erl             kenv            mkdir           ps              rm              sleep           unlink
chmod           df              expr            kill            mv              pwait           rmail           stty            uuidgen
avery@nixphere:~ % 

2.5 Moving around

The cd command stands for 'Change Directory'. Are you seeing the mnemonic conventions yet? like ls, cd can take an argument.

avery@nixphere:~ % cd /usr/bin
avery@nixphere:/usr/bin % pwd
/usr/bin
avery@nixphere:/usr/bin % cd /etc
avery@nixphere:/etc % pwd
/etc
avery@nixphere:/etc % cd ~
avery@nixphere:~ % pwd
/home/avery
avery@nixphere:~ %

2.6 Absolute and relative paths

In UNIX, 'directory' and 'folder' are interchangeable. 'path' means nothing more than 'the path to the directory'. There are two types of paths: absolute and relative. An absolute path starts at the top of the filesystem ('/') and lists all of the subdirectories required to get to the file. A relative path omits everything above our current directory and references subdirectories in our current directory.

Play around with pwd, ls, and cd before continuing.

./ means 'this directory'
../ means 'up a directory'
../../ means 'up two directories'
/ means 'the very top directory'

public_html/ is the relative path to this folder from our current directory
/home/avery/public_html/ is the absolute path to this folder
~ is a shortcut to your home directory 

3. Man Pages

Before we go further, you should know about man pages. the man command is short for 'manual'. All programs installed on your system come with a 'manual' page. To view a man page, we can use the man command. This command places us inside of a separate program called a 'pager'. The pager makes it easier for us to scroll up and down.

avery@nixphere:~ % man ls
LS(1)                   FreeBSD General Commands Manual                  LS(1)

NAME
     ls - list directory contents

SYNOPSIS
     ls [-ABCFGHILPRSTUWZabcdfghiklmnopqrstuwxy1,] [--color=when] [-D format]
        [file ...]

DESCRIPTION
     For each operand that names a file of a type other than directory, ls
     displays its name as well as any requested, associated information.  For
     each operand that names a file of type directory, ls displays the names
     of files contained within that directory, as well as any requested,
     associated information.

     If no operands are given, the contents of the current directory are
     displayed.  If more than one operand is given, non-directory operands are
     displayed first; directory and non-directory operands are sorted
     separately and in lexicographical order.

     The following options are available:

[pager here, press q to quit]
avery@nixphere:~ %

Every manual comes with multiple pages. When you see something like ls(1) , it means the first man page for the ls command. Here is an excerpt from the man page for the man command

The sections of the manual are:
	1.   FreeBSD General Commands Manual
	2.   FreeBSD System Calls Manual
	3.   FreeBSD Library Functions Manual
	4.   FreeBSD Kernel Interfaces Manual
	5.   FreeBSD File Formats Manual
	6.   FreeBSD Games Manual
	7.   FreeBSD Miscellaneous Information Manual
	8.   FreeBSD System Manager's Manual
	9.   FreeBSD Kernel Developer's Manual

Another useful command is apropos . The apropos command is used to search man pages for a specific term.

avery@nixphere:~ % apropos whatis
apropos, whatis(1) - search manual page databases
makewhatis(8) - index UNIX manuals
makewhatis.local(8) - start makewhatis for local file systems
avery@nixphere:~ % 

Notice how apropos tells us which man page to look at. To view the specific page number, we can run:

avery@nixphere:~ % man 8 makewhatis
[ pager here, press q to quit ] 

If you ever need help, be sure to check the man page first before asking online. Some (but not all) man pages contain useful examples too. Developers have spent countless hours documenting their software through manpages. Please use them

4. Files

Everything in UNIX is a file. Well, almost. Everything is represented by a file. Even directories are just a special type of file. Learning file manipulation is an important part of using a UNIX system.

4.1 Special Characters

It can be useful to avoid special characters in your file names. Spaces and characters like ~ ` ^ & * \ / ? | < > all have special meanings in UNIX. If you must use special characters, be sure to surround the entire filename in quotes or use an escape sequence. Escape sequences are a somewhat advanced topic . . . but the most simple way to use them is to prefix a \ character before the character you want to escape. \ essentially means 'hey, ignore whatever character comes after me'. If you don't surround the file name in quotes or use the \ escape sequence for all the special characters in the filename, you will get some unexpected behavior.

avery@nixphere:~ % touch '
Unmatched '''.
avery@nixphere:~ % touch \'
avery@nixphere:~ % ls
'           R           mbox        public_html
avery@nixphere:~ % rm '
Unmatched '''.
avery@nixphere:~ % rm \'
avery@nixphere:~ % ls
R           mbox        public_html
avery@nixphere:~ %

4.1 File Extensions

On windows, file extensions must be three characters or shorter and are mandatory for proper function of the system. Ex: if you don't add .txt you cannot open the file, if you don't add .png you can't open the file.

On UNIX, everything is assumed to be a text file. You don't need to add a .jpg or .txt extension because the system determines what type a file is by actually looking at the data inside the file. You can even cat a .jpg or something to confirm that it's literally just a text file with special information at the beginning and end that tells us to treat it like an image instead of a plain text file. Even programs, you can cat then and see that they are just specially encoded text

For compatability though, be sure to add extensions for anything you serve on the web. Windows is stupid and determines file types not by the file itself but by the extension

4.3 What type of file is this?

the file command tells what type of file we are looking at.

avery@nixphere:~ % ls
R           mbox        public_html
avery@nixphere:~ % file R
R: directory
avery@nixphere:~ % file mbox
mbox: ASCII text
avery@nixphere:~ % file public_html/
public_html/: directory
avery@nixphere:~ % file /dev/mem
/dev/mem: character special (0/38)
avery@nixphere:~ % 

4.4 Creating directories

the mkdir command stands for 'Make Directory'. This command is used to create new folders. the -p flag is useful to create a directory with a subdirectory in it

avery@nixphere:~ % mkdir foo
avery@nixphere:~ % ls
R           foo         mbox        public_html
avery@nixphere:~ % mkdir -p foo/bar/baz
avery@nixphere:~ % ls -aR foo
.   ..  bar

foo/bar:
.   ..  baz

foo/bar/baz:
.  ..
avery@nixphere:~ %

4.5 Deleting directories

The rmdir command stands for 'Remove Directory'. This command can be used to delete empty directories.

If you want to delete a directory that is not empty, you can use the rm -rf command. Please, please, please be careful when using the rm -rf command! If you mistype you can accidentally delete everything! If you must use use this command, consider using rm -ri instead!

avery@nixphere:~ % mkdir foo
avery@nixphere:~ % ls
R           foo         mbox        public_html
avery@nixphere:~ % rmdir foo
avery@nixphere:~ % mkdir foo
avery@nixphere:~ % cp mbox foo/
avery@nixphere:~ % rmdir foo
rmdir: foo: Directory not empty
avery@nixphere:~ % rm -ri foo
remove foo? y
remove foo/mbox? y
avery@nixphere:~ %

4.6 Creating empty files

The touch command is used to create empty files. Don't worry if the file already exists, it won't overwrite it.

avery@nixphere:~ % touch foobar
avery@nixphere:~ % ls
R           foobar      mbox        public_html
avery@nixphere:~ % file foobar
foobar: empty
avery@nixphere:~ % 

4.7 Copying files and directories

The cp command stands for "Copy". It's used to copy files and directories.

avery@nixphere:~ % ls
R           cr          foobar      mbox        public_html
avery@nixphere:~ % cp foobar baz
avery@nixphere:~ % ls
R           baz         cr          foobar      mbox        public_html
avery@nixphere:~ %

If we want to copy a directory, we must pass the -r flag. This stands for 'recursive'.

avery@nixphere:~ % ls
R           cr          mbox        public_html
avery@nixphere:~ % cp R S
cp: R is a directory (not copied).
avery@nixphere:~ % ls
R           cr          mbox        public_html
avery@nixphere:~ % cp -r R S
avery@nixphere:~ % ls
R           S           cr          mbox        public_html
avery@nixphere:~ %

4.8 Moving files and directories

The mv command stands for "Move". It's used to move files and directories. This command is also used to rename files. Note that we don't need the -r flag to move a directory.

avery@nixphere:~ % ls
R           S           cr          foobar      mbox        public_html
avery@nixphere:~ % mv foobar baz
avery@nixphere:~ % ls
R           S           baz         cr          mbox        public_html
avery@nixphere:~ % mv S T
avery@nixphere:~ % ls
R           T           baz         cr          mbox        public_html
avery@nixphere:~ %

4.9 Deleting files and non-empty directories

the rm command stands for "Remove". This command will delete files and directories. If you would like to delete a directory, you should pass the -rf flag.

Be careful, there is no recycle bin on UNIX. If you accidentally delete a file you will need to use disk recovery software. Consider using the -i flag. It will ask for confirmation before deleting a file

avery@nixphere:~ % ls
R           T           baz         cr          mbox        public_html
avery@nixphere:~ % rm baz
avery@nixphere:~ % rm T
rm: T: is a directory
avery@nixphere:~ % rm -rf T
avery@nixphere:~ % ls
R           cr          mbox        public_html
avery@nixphere:~ %

4.10 Printing the contents of a file

the cat command stands for "Concatenate". Although it's primary purpose is for concatenating files, it's also useful for printing the contents on screen.

avery@nixphere:~ % cat /etc/os-release
NAME=FreeBSD
VERSION=13.0-RELEASE-p3
VERSION_ID=13.0
ID=freebsd
ANSI_COLOR="0;31"
PRETTY_NAME="FreeBSD 13.0-RELEASE-p3"
CPE_NAME=cpe:/o:freebsd:freebsd:13.0
HOME_URL=https://FreeBSD.org/
BUG_REPORT_URL=https://bugs.FreeBSD.org/
avery@nixphere:~ % 

4.11 Pagers

While the cat command is useful, it does not provide any scrolling capabilities. If the file we want to read is very big, cat will quickly print the entire file before we even get a change to read it. This is where the less command comes in. less is a program called a pager. It's used for when we want to scroll through a file on our own time.

less is useful for larger files, cat is useful for short files or pipes

Here are some useful shortcuts for less

arrow keys - scrolling
h - open help menu
q - quit less
/ - search for phrase
n - next occurrence of searched phrase
avery@nixphere:~ % less /etc/services
[ pager here ]

5. Editors

Editors are an important part of any UNIX system. Since everything in UNIX is a file, most system settings are actually stored in a plain text file somewhere on the system. There are many editors to choose from. We will go over a few (in order of learning curve). You should try all available editors.

5.1 Nano

Nano is an easy to use editor. It's an improved version of an old editor called Pico. Nano even has a friendly menu at the bottom of the screen that reminds you of shortcuts.

GNU nano 5.8                       New Buffer                       Modified  
type some text here




















^G Help      ^O Write Out ^W Where Is  ^K Cut       ^T Execute   ^C Location
^X Exit      ^R Read File ^\ Replace   ^U Paste     ^J Justify   ^/ Go To Line

5.2 ee

ee is a another easy to use editor. It's not standard on GNU systems but comes pre installed on FreeBSD.

^[ (escape) menu ^y search prompt ^k delete line   ^p prev li     ^g prev page
^o ascii code    ^x search        ^l undelete line ^n next li     ^v next page
^u end of file   ^a begin of line ^w delete word   ^b back 1 char ^z next word
^t top of text   ^e end of line   ^r restore word  ^f forward char
^c command       ^d delete char   ^j undelete char              ESC-Enter: exit
=====line 1 col 0 lines from top 1 ============================================
type some text here
















no file

5.3 Vi

Vi stands for 'visual editor'. There is a fork called Vim that stands for 'vi improved'. Many people prefer Vim over Vi because of it's increased feature set. On every UNIX, Vi or vim will be installed by default.

Vi(m) has a fairly high learning curve but it's the most efficient editor on earth. It uses concepts like modal editing, ergonomic shortcuts, and a command interface. It supports macros, syntax highlighting, auto indent, search and replace, etc. If you have the time to invest, Vi(m) is definitely worth Learning.

Some useful shortcuts:

h,j,k,l - These are your cursor keys. in order, they are left, down, up, and right.

i - this will place you into insert mode. You can type as normal

Escape - this will place you into normal mode. This mode is used for moving around

: - this will place you into command mode

:wq - Save and quit file (aka Write Quit)
type some text here
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
"file" 1L, 20B                                               1,19          All

5.4 Other editors

There are many other editors available on a UNIX system. Ed is the standard line editor, Joe is similar to nano, Jed has pseudo menus, Micro is a terminal based editor with 'modern' (aka familiar) keyboard shortcut (ex: Control + c, Control + v), and Emacs is an editor that doesn't work in POSIX compliant terminals.

6. Regex

'Regex' is short for 'Regular Expressions'. Some people call these 'wildcards'. Let's look at some basic regex.

* - match zero or more characters
? - match a single character
[] - match a range of characters
avery@nixphere:~ % ls /bin/*sh*
/bin/csh  /bin/sh   /bin/tcsh
avery@nixphere:~ % ls /bin/?sh
/bin/csh
avery@nixphere:~ % ls /bin/*[a-f]
/bin/chmod      /bin/date       /bin/dd         /bin/df         /bin/domainname /bin/ed         /bin/hostname   /bin/pwd        /bin/red        /bin/sync
avery@nixphere:~ %

Let's look at more practical examples. If we want to list all files with a specific extension, we can use *.extension

avery@nixphere:~ % ls *.png
android-chrome-192x192.png android-chrome-512x512.png apple-touch-icon.png       favicon-16x16.png          favicon-32x32.png

If we want to list files with a specific numeber of characters in the name, we can do this:

avery@nixphere:~ % ls ?????.*
index.php

What if we want to find all files with numbers in the name?

avery@nixphere:~ % ls *[0-9]*
android-chrome-192x192.png android-chrome-512x512.png favicon-16x16.png          favicon-32x32.png

7. Users and Permissions

7.1 Users, Groups, and Permissions explained

UNIX is a multi user, time sharing operating system. This means that multiple users can be logged in to one machine at the same time. In contrast to windows where ther is effectively only one user at any given time, a UNIX system has many users and 'Daemons' running concurrently. A daemon is a special type of user that runs a server or process (ex sshd runs under the ssh daemon, httpd runs under the http daemon, etc). Additionally, on UNIX there is no 'Administrator' 'account'. The UNIX admin user is called 'root'. 'root' is a real account that has unrestricted access to the entire system. In order to avoid breaking things and creating additional vulnerabilities, many sysadmins choose to install a program called sudo . Sudo is similar to the windows concept of "Right Click -> Run as Administrator". sudo supports logging, restrictions on which commands a user can run, how a user must authenticate to run a program as root, etc.

Alongside users, UNIX has groups. A group is used to grant permissions or access to a specific list of accounts. For example: Accounts in the 'wheel' or 'sudoers' groups are allowed to use the sudo command to run commands as root. Users in the 'daemon' group are system accounts and are prevented from logging in as a normal user.

In a real world example, everyone who works in the Accounting department would be in the 'accounting' group. They all have access to the files related to accounting. The Data Entry department does not need to access the documents and files owned by the Accounting department so we would want to restrict their access. Likewise, the users in the Developer department do not need access to the Accounting department's files but they do need access to a communal workspace (owned by the 'developer' group) where they can all work together. The IT department does not need to access what the Accounting department is working on but they all need to share files related to network documentation and the ticket tracker as well as whatever the Development department is working on. A user in the IT department might be in both the 'developer' group and the 'it' group. Groups are useful because they allow us to group users together in a way that allows them to share resources while preventing them from accessing resources they have no need to ever touch. A user can be in multiple groups at once. This makes the entire concept of permissions extremely flexible.

In addition to sudo and groups, the designers of UNIX created a permission to prevent users from harming each other (or the system). The permission system is quite simple.

7.2 Listing permissions

Every file is owned by a User and a Group . Each file has 3 permissions: Read, Write, and Execute. These permissions can be different for the User , the Group and Other Users .

These 3 permissions define what can be done with the file. If the Read bit is set for User, Group, Other then all users on the system can read the file. If the Write bit is set for User, Group , then the user and group of the file can edit or overwrite that file. If the Execute bit is set for only the User , then the user who owns that file can run the file as if it was any other program.

Be careful when playing with permissions! If you accidentially allow other users to modify your or ready files they can destroy or exfiltrate your data!

Let's take a look at some files. In order to get a list of permissions, you can run the ls -lh command.

In order to explain the output of ls -lh , I have provided a diagram

avery@nixphere:~ % ls -lh /bin/sh
-r-xr-xr-x  1 root  wheel   159K Apr  8 23:17 /bin/sh
|             |     |       |     |        |   |
|            User  Group   Size   |        |  File Name
|                                 |        |
Permissions                     Date last edited

The permissions block can be somewhat confusing at first. Let's diagram further

d r w x r w x r w x
| | | | | | | | | |
| | | | | | | | | Other execute permission
| | | | | | | | Other write permission
| | | | | | | Other read permission
| | | | | | Group execute permission
| | | | | Group write permission
| | | | Group read permission
| | | User execute permission
| | User wirte permission
| User read permission 
Indicates directory

the most simple way to remember the diagram from above is by remembering the 'UGO' acronym. This stands for 'User, Group, Other'. We can group the permissions together in sets of three to reduce the number of things we need to remember.

-[rwx][r-x][r-x]
  |    |    |
  |    |    Other permissions
  |    Group permissions
  User permissions

7.3 Chaging permissions

In order to change permissions, we can use the chmod command. chmod is short for 'Change file Mode'.

To use chmod , you list which set to change permissions for (UGOA), the + or - symbols to add or remove a permissions, and list of permissions to give or revoke (RWX). We can combine operations and set permissions for multiple sets at once.

An easy way to remember how to use chmod is to remember two acronyms: 'UGOA' and 'RWX'. These stand for 'User, Group, Other, All' and 'Read, Write, Execute'.

avery@nixphere:~ % ls -lh foobar
-rw-r--r--  1 avery  avery     0B Aug  8 19:14 foobar
avery@nixphere:~ % chmod u+x ./foobar
avery@nixphere:~ % ls -lh foobar
-rwxr--r--  1 avery  avery     0B Aug  8 19:14 foobar
avery@nixphere:~ % chmod og-r ./foobar
avery@nixphere:~ % ls -lh foobar
-rwx------  1 avery  avery     0B Aug  8 19:14 foobar
avery@nixphere:~ % chmod a+rwx ./foobar
avery@nixphere:~ % ls -lh foobar
-rwxrwxrwx  1 avery  avery     0B Aug  8 19:14 foobar
avery@nixphere:~ % 

7.4 Chaging Ownership

The chown command stands for "Change Ownership" and is used for changing which user and group owns a file. This command will not work in many cases unless you are allowed to run the sudo command. The chown command changes ownership in the form of user:group

avery@nixphere:~ % ls -lh foobar
-rwxrwxrwx  1 avery  avery     0B Aug  8 19:26 foobar
avery@nixphere:~ % chown avery:wheel ./foobar
avery@nixphere:~ % ls -lh foobar
-rwxrwxrwx  1 avery  wheel     0B Aug  8 19:26 foobar
avery@nixphere:~ % sudo chown root:www ./foobar
Password:
avery@nixphere:~ % ls -lh ./foobar
-rwxrwxrwx  1 root  www     0B Aug  8 19:26 ./foobar
avery@nixphere:~ %

8. Grep

grep is a tool that's used for filtering text. Think of grep as a search function. To use grep , specify a phrase you want to search for then the file you want to search in.

avery@nixphere:~ % grep "ssh" /etc/services
ssh		 22/tcp	   #Secure Shell Login
ssh		 22/udp	   #Secure Shell Login
ssh		 22/sctp   #Secure Shell Login
sshell		614/tcp	   #SSLshell
sshell		614/udp
netconf-ssh	830/tcp    #NETCONF over SSH
netconf-ssh	830/udp    #NETCONF over SSH
x11-ssh		6010/tcp   #Unofficial name, for convenience
x11-ssh		6010/udp
avery@nixphere:~ % grep "avery" /etc/passwd
avery:*:1002:1003:Avery:/home/avery:/bin/tcsh
avery@nixphere:~ % 

To find all lines that do not contain a specific string, we can use the -v flag.

avery@nixphere:~ % cat /etc/os-release
NAME=FreeBSD
VERSION=13.0-RELEASE-p3
VERSION_ID=13.0
ID=freebsd
ANSI_COLOR="0;31"
PRETTY_NAME="FreeBSD 13.0-RELEASE-p3"
CPE_NAME=cpe:/o:freebsd:freebsd:13.0
HOME_URL=https://FreeBSD.org/
BUG_REPORT_URL=https://bugs.FreeBSD.org/
avery@nixphere:~ % grep -v "FreeBSD" /etc/os-release
VERSION=13.0-RELEASE-p3
VERSION_ID=13.0
ID=freebsd
ANSI_COLOR="0;31"
CPE_NAME=cpe:/o:freebsd:freebsd:13.0
avery@nixphere:~ %

To search case insensitively, we can use the -i flag.

avery@nixphere:~ % grep -i ansi /etc/os-release 
ANSI_COLOR="0;31"
avery@nixphere:~ % 

to search for multiple phrases, we can use the -e flag. Try combining flags to see what you can do. Be sure to surround your phrases in quotes if they contain any spaces or special characters.

avery@nixphere:~ % grep -e bug -i -e id /etc/os-release
VERSION_ID=13.0
ID=freebsd
BUG_REPORT_URL=https://bugs.FreeBSD.org/
avery@nixphere:~ % 

9. Pipes

9.1 Data Streams

Every program you can run has three data streams:

STDIN - Standard Input, the data fed to the program

STDOUT - Standard Output, data printed to the terminal by the program

STDERR - Standard Error, Error messages (printed to the terminal by default).

UNIX pipes allow us to redirect these data streams in useful ways. Nearly every UNIX command supports piping. With pipes we can redirect command output to a file, redirect input from a file, and even use the output of one program as input for another program. Using one command's output as input for another program allows us to chain together many commands in order to create programs of various complexity.

Here are the pipes. They allow us to do specific operations

>	save command output to a file
>>	append output to a file
<	read input from a file
2>	redirect error messages
|	send the output of one program to the input of another

For now, you can ignore standard error. The way to redirect STDERR is not consistent across shells. 2> works in bourne-compatible shells but not in every shell.

9.2 Using Pipes

Once you become familiar with pipes you'll feel a bit strange when working in an environment without them. Let's play with them.

Here we are redirecting output. Notice how it's identical to what's been printed.

avery@nixphere:~ % ls
R           cr          mbox        procs.svg   public_html
avery@nixphere:~ % ls > files
avery@nixphere:~ % cat files
R
cr
files
mbox
procs.svg
public_html
avery@nixphere:~ %

Let's append more data to this file we've created. If we were to use the > pipe again, we'd overwrite the file. So let's use the >> pipe instead.

avery@nixphere:~ % ls -d .* >> files
avery@nixphere:~ % cat files
R
cr
files
mbox
procs.svg
public_html
.
..
.Rhistory
.config
.cshrc
.history
.local
.login
.login_conf
.mail_aliases
.mailrc
.profile
.shrc
.ssh
.vim
.viminfo
avery@nixphere:~ %

Now let's use the < pipe to read a file into a program. In this use case, less < files is equivalent to less files

avery@nixphere:~ % less < files
R
cr
files
mbox
procs.svg
public_html
.
..
.Rhistory
.config
.cshrc
.history
.local
.login
.login_conf
.mail_aliases
.mailrc
.profile
.shrc
.ssh
.vim
.viminfo
(END)

Now we can try out the pipe operator. The | character is called 'pipe' and is usually on the same key as the \ character. Below are some fun examples. Cowsay and figlet are rarely used in serious business but they provide a good example of how pipes work. Try catting a file and piping it through cowsay.

avery@nixphere:~ % cat files | grep "." | grep rc | figlet -f small
         _           
   __ __| |_  _ _ __ 
 _/ _(_-< ' \| '_/ _|
(_)__/__/_||_|_| \__|
                     
              _ _        
   _ __  __ _(_) |_ _ __ 
 _| '  \/ _` | | | '_/ _|
(_)_|_|_\__,_|_|_|_| \__|
                         
      _           
   __| |_  _ _ __ 
 _(_-< ' \| '_/ _|
(_)__/_||_|_| \__|
                  
avery@nixphere:~ % cat files | grep "." | grep rc | cowsay
 ______________________ 
< .cshrc .mailrc .shrc >
 ---------------------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
avery@nixphere:~ % 

You can combine pipes and redirection too. Try to redirect the output of cowsay to a file, then pipe that same file into cowsay. See what happens.

One tip for preventing pipe problems is to build your pipes incrementally. Add another pipe only when you have achieved a result you're looking for.

10. Filters

Now that you know about pipes, filters will be useful. In UNIX, a filter command takes whatever data you give it, performs some operation, then outputs the modified data. Be sure to read the man pages for each command. These examples do not demonstrate the full capabilities of each command.

10.1 head

the head command is used for displaying the first few lines of a file. Using the -n flag specifies how many lines to display. By defualt, 10 lines are displayed.

avery@nixphere:~ % head /etc/protocols
#
# Internet protocols
#
# $FreeBSD$
#	from: @(#)protocols	5.1 (Berkeley) 4/17/89
#
# See also http://www.iana.org/assignments/protocol-numbers
#
ip	0	IP		# internet protocol, pseudo protocol number
#hopopt	0	HOPOPT		# hop-by-hop options for ipv6
avery@nixphere:~ % head -n 5 /etc/protocols
#
# Internet protocols
#
# $FreeBSD$
#	from: @(#)protocols	5.1 (Berkeley) 4/17/89
avery@nixphere:~ % 

10.2 tail

The tail command works similarly to the head command. You can also use the -n flag to specify any number of lines. Another useful flat is the -f flag. This flag will continually print lines as they are added to the file until you press Control + c. tail -f is useful for watching log files in real time.

avery@nixphere:~ % tail /etc/protocols
manet	138	MANET		# MANET Protocols (RFC5498)
hip	139	HIP		# Host Identity Protocol (RFC5201)
shim6	140	SHIM6		# Shim6 Protocol (RFC5533)
wesp	141	WESP		# Wrapped Encapsulating Security Payload (RFC5840)
rohc	142	ROHC		# Robust Header Compression (RFC5858)
#	138-254			# Unassigned
pfsync	240	PFSYNC		# PF Synchronization
#	253-254			# Use for experimentation and testing (RFC3692)
#	255			# Reserved
divert	258	DIVERT		# Divert pseudo-protocol [non IANA]
avery@nixphere:~ % tail -n 3 /etc/protocols
#	253-254			# Use for experimentation and testing (RFC3692)
#	255			# Reserved
divert	258	DIVERT		# Divert pseudo-protocol [non IANA]
avery@nixphere:~ % 

10.3 sort

The sort command does what it sounds like: sort the file by lines.

avery@nixphere:~ % cat animals
dog
cat
hippo
duck
penguin
shark
avery@nixphere:~ % cat animals | sort
cat
dog
duck
hippo
penguin
shark
avery@nixphere:~ % cat animals | sort > animals.sorted
avery@nixphere:~ % cat animals.sorted
cat
dog
duck
hippo
penguin
shark
avery@nixphere:~ % 

The sort command can also sort by numbers. Use the -n flag to sort by numeric order instead of alphabetically.

avery@nixphere:~ % cat animals
2 moose
0 pegasus
4 antelope
6 giraffe
3 walrus
1 snake
avery@nixphere:~ % cat animals | sort -n
0 pegasus
1 snake
2 moose
3 walrus
4 antelope
6 giraffe
avery@nixphere:~ % cat animals | sort -n > animals.sorted
avery@nixphere:~ % cat animals.sorted 
0 pegasus
1 snake
2 moose
3 walrus
4 antelope
6 giraffe
avery@nixphere:~ % 

10.4 wc

The wc command stands for "Word Count". This command can be used to count lines, words, or characters in a file. The -l flag means "Lines", the -m flag means "characters", and the -w flag means "Words".

avery@nixphere:~ % cat animals
ostrich
penguin puffin
pheasant grouse
otter
trout
goose
avery@nixphere:~ % cat animals | wc -l
       6
avery@nixphere:~ % cat animals | wc -m
      57
avery@nixphere:~ % cat animals | wc -w
       8
avery@nixphere:~ % 

10.5 nl

The nl command stands for "Number Lines". This command is useful for when you want to number the lines in your file.

avery@nixphere:~ % cat animals
ostrich
penguin puffin
pheasant grouse
otter
trout
goose
avery@nixphere:~ % cat animals | nl
     1	ostrich
     2	penguin puffin
     3	pheasant grouse
     4	otter
     5	trout
     6	goose
avery@nixphere:~ % cat animals | nl > animals.numbered
avery@nixphere:~ % cat animals.numbered 
     1	ostrich
     2	penguin puffin
     3	pheasant grouse
     4	otter
     5	trout
     6	goose
avery@nixphere:~ % 

10.6 uniq

The uniq command is used for removing duplicates or printing only duplicates. Notice how cat appears twice.

avery@nixphere:~ % cat animals 
cat
wolf
salmon
trout
cat
bear
bear
goose
avery@nixphere:~ % cat animals | uniq
cat
wolf
salmon
trout
cat
bear
goose

uniq works best when we pass our data through sort first. Be sure to add a sort command before uniq if you want accurate results.

avery@nixphere:~ % cat animals | sort | uniq
bear
cat
goose
salmon
trout
wolf

The -d flag means "print only duplicate lines".

avery@nixphere:~ % cat animals | sort | uniq -d
bear
cat
avery@nixphere:~ % 

10.7 tac

The tac command is just the cat command but backwards. Instead of printing the file in order, it will print it in reverse. If tac is not available on your system, you can use tail -r instead.

avery@nixphere:~ % cat animals 
cat
wolf
salmon
trout
cat
bear
bear
goose
avery@nixphere:~ % tac animals 
goose
bear
bear
cat
trout
salmon
wolf
cat
avery@nixphere:~ % tail -r animals 
goose
bear
bear
cat
trout
salmon
wolf
cat
avery@nixphere:~ % 

10.8 shuf

The shuf command stands for "Shuffle". This command will shuflfe the lines in your file. If shuf is not available on your system, you can use sort --random-sort instead.

avery@nixphere:~ % cat animals
cat
wolf
salmon
trout
cat
bear
bear
goose
avery@nixphere:~ % cat animals | shuf
trout
goose
cat
cat
salmon
bear
wolf
bear
avery@nixphere:~ % cat animals | shuf
trout
cat
bear
goose
cat
bear
wolf
salmon
avery@nixphere:~ % cat animals | sort --random-sort
bear
bear
salmon
cat
cat
trout
goose
wolf
avery@nixphere:~ % 

10.9 rs

The rs command is used for converting a file with many lines into a format that's more friendly for reading into a data array.

avery@nixphere:~ % cat animals 
cat
wolf
salmon
trout
cat
bear
bear
goose
avery@nixphere:~ % cat animals | rs
cat     wolf    salmon  trout   cat     bear    bear    goose
avery@nixphere:~ % 

10.10 tr

The tr command stands for "Translate". It is used for translating on specific character (or phrase) into another.

avery@nixphere:~ % cat animals
cat
wolf
salmon
trout
cat
bear
bear
goose
avery@nixphere:~ % cat animals | tr cat bat
bat
wolf
salmon
trout
bat
bear
bear
goose
avery@nixphere:~ % cat animals | tr "\n" " " && printf "\n"
cat wolf salmon trout cat bear bear goose 
avery@nixphere:~ % 

10.11 cut

The cut command is useful for cutting text at a delimiter. It is most useful for data formatted as a .csv file or as a matrix. Use the -d flag to specify a delimiter and the -f flag to specify a field

avery@nixphere:~ % cat animals.csv 
"BIRDS", "FELINES", "CANINES", "FISH"
"pigeon", "ocelot", "wolf", "trout"
"toucan", "tiger", "labrador", "salmon"
"parrot", "lion", "poodle", "swordfish"
"sparrow", "cheetah", "english bulldog", "tuna"
"bluejay", "leopard", "malamute", "sardine" 
avery@nixphere:~ % cat animals.csv | cut -d , -f 1
"BIRDS"
"pigeon"
"toucan"
"parrot"
"sparrow"
"bluejay"
avery@nixphere:~ % cat animals.csv | cut -d , -f 2
 "FELINES"
 "ocelot"
 "tiger"
 "lion"
 "cheetah"
 "leopard"
avery@nixphere:~ % cat animals.csv | cut -d , -f 4
 "FISH"
 "trout"
 "salmon"
 "swordfish"
 "tuna"
 "sardine" 
avery@nixphere:~ % 

10.12 sed

The sed command stands for "Stream EDitor". It's one of the most useful commands you could possibly learn but has somewhat of a learning curve. Although sed can do some very advanced things, it's most useful for search and replace. Typically, you will run something like:

cat $data | sed s/search/replace/g 

Here are some examples

avery@nixphere:~ % cat animals.csv
"BIRDS", "FELINES", "CANINES", "FISH"
"pigeon", "ocelot", "wolf", "trout"
"toucan", "tiger", "labrador", "salmon"
"parrot", "lion", "poodle", "swordfish"
"sparrow", "cheetah", "english bulldog", "tuna"
"bluejay", "leopard", "malamute", "sardine" 
avery@nixphere:~ % cat animals.csv | sed s/\"/\'/g
'BIRDS', 'FELINES', 'CANINES', 'FISH'
'pigeon', 'ocelot', 'wolf', 'trout'
'toucan', 'tiger', 'labrador', 'salmon'
'parrot', 'lion', 'poodle', 'swordfish'
'sparrow', 'cheetah', 'english bulldog', 'tuna'
'bluejay', 'leopard', 'malamute', 'sardine' 
avery@nixphere:~ % cat animals.csv | sed s/\"//g | sed s/", "/" comma "/g
BIRDS comma FELINES comma CANINES comma FISH
pigeon comma ocelot comma wolf comma trout
toucan comma tiger comma labrador comma salmon
parrot comma lion comma poodle comma swordfish
sparrow comma cheetah comma english bulldog comma tuna
bluejay comma leopard comma malamute comma sardine 
avery@nixphere:~ % 

10.13 awk

The awk command is like sed but more advanced. If you are dedicated enough, awk can be used a complete programming language. In most cases though, awk is used to achieve a result similar to cut.

avery@nixphere:~ % cat tally
PERSON ITEM QUANTITY
Alice bananas 21
Elizabeth potatoes 8lbs
Jean-Luc replicatorBurgers 2
Ollie oranges 4
Emme bananas 5
Phillip potatoes 2
Lennon lemons 2
avery@nixphere:~ % cat tally | awk '{print $1 " - " $2 " : " $3 }'
PERSON - ITEM : QUANTITY
Alice - bananas : 21
Elizabeth - potatoes : 8lbs
Jean-Luc - replicatorBurgers : 2
Ollie - oranges : 4
Emme - bananas : 5
Phillip - potatoes : 2
Lennon - lemons : 2
avery@nixphere:~ % cat tally | awk '{print $1 " purchased " $3 " " $2 }' | grep -v PERSON
Alice purchased 21 bananas
Elizabeth purchased 8lbs potatoes
Jean-Luc purchased 2 replicatorBurgers
Ollie purchased 4 oranges
Emme purchased 5 bananas
Phillip purchased 2 potatoes
Lennon purchased 2 lemons
avery@nixphere:~ % 

awk is also useful for formatting data so that it can be read by other programs. Here we are converting our plaintext data to a CSV

avery@nixphere:~ % cat tally
PERSON ITEM QUANTITY
Alice bananas 21
Elizabeth potatoes 8lbs
Jean-Luc replicatorBurgers 2
Ollie oranges 4
Emme bananas 5
Phillip potatoes 2
avery@nixphere:~ % cat tally | awk '{print "\"" $1 "\"," "\"" $2 "\"," "\"" $3 "\""  }'
"PERSON","ITEM","QUANTITY"
"Alice","bananas","21"
"Elizabeth","potatoes","8lbs"
"Jean-Luc","replicatorBurgers","2"
"Ollie","oranges","4"
"Emme","bananas","5"
"Phillip","potatoes","2"
"Lennon","lemons","2"
avery@nixphere:~ % cat tally | awk '{print "\"" $1 "\"," "\"" $2 "\"," "\"" $3 "\""  }' > tally.csv
avery@nixphere:~ % cat tally.csv
"PERSON","ITEM","QUANTITY"
"Alice","bananas","21"
"Elizabeth","potatoes","8lbs"
"Jean-Luc","replicatorBurgers","2"
"Ollie","oranges","4"
"Emme","bananas","5"
"Phillip","potatoes","2"
"Lennon","lemons","2"
avery@nixphere:~ % 

10.14 diff

The diff command is mostly used by developers. The diff tool can be used to show how two files are different. If no output is shown, the files are the same. If we redirect the output of the diff command to a file we can create a patch. We can then email our patch around and other users can apply it to their files in order to benefit from our improvements. The patch command automatically creates a backup of your file with a .orig extension so don't worry about accidentally adding a bad patch because you can always go back.

avery@nixphere:~ % cat script-v1.sh
#!/bin/sh
echo "hello world"
echo "here is a list of files: "
ls
avery@nixphere:~ % cat script-v2.sh 
#!/bin/sh
echo "Hello World"
echo "here is a list of files: "
ls -l
avery@nixphere:~ % diff script-v1.sh script-v2.sh 
2c2
< echo "hello world"
---
> echo "Hello World"
4c4
< ls
---
> ls -l
avery@nixphere:~ % diff script-v1.sh script-v2.sh > patch
avery@nixphere:~ % patch script-v1.sh patch
Hmm...  Looks like a normal diff to me...
Patching file script-v1.sh using Plan A...
Hunk #1 succeeded at 2.
Hunk #2 succeeded at 4.
done
avery@nixphere:~ % diff script-v1.sh script-v2.sh 
avery@nixphere:~ % 

10.15 hd

The hd command stands for "Hex Dump". This is another command that's really only useful if you're a developer.

avery@nixphere:~ % cat script-v1.sh
#!/bin/sh
echo "hello world"
echo "here is a list of files: "
ls
avery@nixphere:~ % cat script-v1.sh | hd
00000000  23 21 2f 62 69 6e 2f 73  68 0a 65 63 68 6f 20 22  |#!/bin/sh.echo "|
00000010  68 65 6c 6c 6f 20 77 6f  72 6c 64 22 0a 65 63 68  |hello world".ech|
00000020  6f 20 22 68 65 72 65 20  69 73 20 61 20 6c 69 73  |o "here is a lis|
00000030  74 20 6f 66 20 66 69 6c  65 73 3a 20 22 0a 6c 73  |t of files: ".ls|
00000040  0a                                                |.|
00000041
avery@nixphere:~ % 

11. Escape Sequences

From Wikipedia:

ANSI escape sequences are a standard for in-band signaling to control cursor location, color, font styling, and other options on video text terminals and terminal emulators. Certain sequences of bytes, most starting with an ASCII escape character and a bracket character, are embedded into text. The terminal interprets these sequences as commands, rather than text to display verbatim.

ANSI sequences were introduced in the 1970s to replace vendor-specific sequences and became widespread in the computer equipment market by the early 1980s. They are used in development, scientific, commercial text-based applications as well as bulletin board systems to offer standardized functionality.

ANSI sequences were introduced in the 1970s to replace vendor-specific sequences and became widespread in the computer equipment market by the early 1980s. They are used in development, scientific, commercial text-based applications as well as bulletin board systems to offer standardized functionality. Although hardware text terminals have become increasingly rare in the 21st century, the relevance of the ANSI standard persists because a great majority of terminal emulators and command consoles interpret at least a portion of the ANSI standard.

Or more simply, escape sequences are a combination of keys you can press or characters you can type to instruct the terminal to do something specific. An escape seqence is a series of characters that the terminal interprets as a single special control character.

11.1 Useful Shortcuts

Here are some useful keyboard shortcuts. When reading the table, assume that ^ means "Press the control key and whatever key comes after it at the same time".

Control + key Description
^C Cancel running program
^D logout (runs exit command )
^L Clear screen
^E Move cursor to end of line
^A Move cursor ot beginning of line
^P or Up Arrow Scroll upwards through command history
^N or Down Arrow Scroll downwards through command history
TAB or ^I Command autocompletion (press multiple times)
^R Search through command history (doesn't work on all shells)

11.2 ANSI Control Sequences

These escape sequences are useful for when you need to type a specific character when your keyboard doesn't support it. When reading the table, assume that ^ means "Press the control key and whatever key comes after it at the same time". If you want to escape these characters for a command, type \ then the key. These are additional special characters. You should escape them when writing scripts or trying to use a command like sed to replace newlines or tabs.

\t means "hey terminal, this is a tab character"
Control + i means "hey terminal, print a tab character right now" 
Escape Control + key Description
\a ^G Terminal Bell (make it beep)
\b ^H Backspace
\t ^I Tab
\n ^J Newline aka enter key
\f ^L Clear Screen
\r ^M Carrige return aka enter key
\e ^[ Escape key

11.3 ANSI Color Codes

You can use echo -e to print these escape sequences to the ternimal. If the -e flag is not available on your POSIX version of echo , you can use the printf command instead. These two commands are very simple in that they will print exactly what you tell them to the terminal.

ForegroundBackgroundName
3040Black
3141Red
3242Green
3343Yellow
3444Blue
3545Magenta
3646Cyan
3747White

How you type ANSI color codes will vary depending on your system. Typically, you will type one of the following prefixes followed by a [ character, the color code, them the m character. If you want to use multiple color codes (eg foreground and background at the same time), put a ; character between them. I personally find \033 the easiest to type.

Control + v then Escape
\033
\u001b
\x1b

Let's play with some color codes. Be sure to end your command with \033[0m to reset your terminal to it's default colors. If you experience any issues, you can run the reset command to fix your terminal. Also notice how I end each printf command with \n , the escape for a newline character.

Setting the foreground color:

avery@nixphere:~ % printf '\033[31mthis is red\033[0m\n'
this is red
avery@nixphere:~ % printf '\033[36mthis is cyan\033[0m\n'
this is cyan
avery@nixphere:~ % printf '\033[33mthis is yellow\033[0m\n'
this is yellow
avery@nixphere:~ %

Setting the background color:

avery@nixphere:~ % printf '\033[41mthis is red\033[0m\n'
this is red
avery@nixphere:~ % printf '\033[44mthis is blue\033[0m\n'
this is blue
avery@nixphere:~ % printf '\033[42mthis is green\033[0m\n'
this is green
avery@nixphere:~ %

Setting both the foreground and background color:

avery@nixphere:~ % printf '\033[34;41mblue on red\033[0m\n'
blue on red
avery@nixphere:~ % printf '\033[30;47mblack on white\033[0m\n'
black on white
avery@nixphere:~ % printf '\033[35;46mmagenta on cyan\033[0m\n'
magenta on cyan
avery@nixphere:~ % 

11.4 ANSI Select Graphic Rendition Codes

These escape sequences can be used for formatting text in special ways other than colors. Some of these might not be supported in your terminal emulator. Assume that everything past escape code 55 is unsupported and that everything else is rarely supported. If you must use one of these escape sequences, stick with 0-9 and 21-41. If you want to ensure that your escape sequence will work everywhere, use only bold, underline, and inverted.

Code What it does
0 Reset or normal
1 Bold or increased intensity
2 Faint, decreased intensity, or dim
3 Italic
4 Underline
5 Slow blink
6 Rapid blink
7 Reverse video or invert
8 Conceal or hide
9 Crossed-out, or strike
10 Primary (default) font
11-19 Alternative font
20 Blackletter font
21 Doubly underlined; or: not bold
22 Normal intensity
23 Neither italic, nor blackletter
24 Not underlined
25 Not blinking
26 Proportional spacing
27 Not reversed
28 Reveal
29 Not crossed out
30-37 Set foreground color
38 Set foreground color
39 Default foreground color
40-47 Set background color
48 Set background color
49 Default background color
50 Disable proportional spacing
51 Framed
52 Encircled
53 Overlined
54 Neither framed nor encircled
55 Not overlined
58 Set underline color
59 Default underline color
60 Ideogram underline or right side line
61 Ideogram double underline, or double line on the right side
62 Ideogram overline or left side line
63 Ideogram double overline, or double line on the left side
64 Ideogram stress marking
65 No ideogram attributes
73 Superscript
74 Subscript
75 Neither superscript nor subscript
90-97 Set bright foreground color
100-107 Set bright background color
avery@nixphere:~ % printf "\033[1m bold \033[2m faint \033[0m normal \n"               
 bold  faint  normal
avery@nixphere:~ % printf "\033[4m underlined \033[24;53m overlined \033[55m neither \033[4;53m both \033[0m\n"
 underlined  overlined  neither  both 
avery@nixphere:~ % printf "\033[5m blinking \033[0m\n"                   
 blinking 
avery@nixphere:~ % printf "\033[7m inverted \033[0m\n"
 inverted 
avery@nixphere:~ % 

12. Processes

On UNIX, everything that isn't a file is a process. A process is any command running on the system. If you've ever opened Task Manager on windows, you were viewing various processes running on the system. Learning process management is an important skill for anyone spending time in the shell

12.1 ps

The ps command displays processes running on the system. ps stands for "ProcesS". By default, ps will only show commands that your user is running and have a controlling terminal. The most useful flags for ps are the -a flag which stands for "All users", the -u flag which shows additional information, and the -x flag which displays processes that do not have a controlling terminal. Combining all these flags together, we can see all the processes running on the system by running ps aux .

avery@nixphere:~ % ps
  PID TT  STAT    TIME COMMAND
32216  0  INs  0:00.05 -tcsh (tcsh)
32272  2  SNs  0:00.18 -tcsh (tcsh)
32603  2  RN+  0:00.00 ps
avery@nixphere:~ % ps ux
USER    PID %CPU %MEM   VSZ  RSS TT  STAT STARTED    TIME COMMAND
avery 32215  0.0  0.4 21396 8704  -  IN   23:52   0:02.02 sshd: avery@pts/0 (sshd)
avery 32271  0.0  0.4 21396 8664  -  SN   23:55   0:00.21 sshd: avery@pts/2 (sshd)
avery 32216  0.0  0.2 13984 3828  0  INs  23:52   0:00.05 -tcsh (tcsh)
avery 32272  0.0  0.2 13984 4076  2  SNs  23:55   0:00.18 -tcsh (tcsh)
avery 32604  0.0  0.1 13472 2580  2  RN+  00:54   0:00.00 ps ux
avery@nixphere:~ % ps aux
[ very long list of processes here ]

The most important thing ps tells us can be found in the "PID" column. "PID" stands for Process ID. We can use a process' PID to manipulate it.

12.2 top

top is a command that displays a more interactive verison of ps . I personally do not like top because I prefer ps aux | grep [what I am looking for ]. Some people use htop instead but htop is bloat.

avery@nixphere:~ % top
last pid: 32630;  load averages:  0.64,  0.65,  0.67   up 16+00:16:52  00:58:26
72 processes:  2 running, 70 sleeping
CPU:  2.9% user,  0.0% nice,  1.0% system,  0.2% interrupt, 96.0% idle
Mem: 76M Active, 106M Inact, 64M Laundry, 1368M Wired, 319M Free
ARC: 802M Total, 276M MFU, 353M MRU, 132K Anon, 14M Header, 158M Other
     527M Compressed, 1117M Uncompressed, 2.12:1 Ratio
Swap: 4096M Total, 235M Used, 3861M Free, 5% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME    WCPU COMMAND
45149 pleroma      18  52    0  1227M    72M select 344:09   1.95% beam.smp
45317 postgres      1  20    0   183M    36M select  32:13   0.00% postgres
45319 postgres      1  20    0   183M    36M select  32:10   0.00% postgres
45318 postgres      1  20    0   183M    36M select  32:08   0.00% postgres
45320 postgres      1  20    0   183M    37M select  32:00   0.00% postgres
45323 postgres      1  20    0   183M    37M select  32:00   0.00% postgres
45321 postgres      1  20    0   183M    36M select  32:00   0.00% postgres
45322 postgres      1  20    0   183M    37M select  32:00   0.00% postgres
45324 postgres      1  20    0   183M    37M select  31:58   0.00% postgres
45326 postgres      1  20    0   183M    37M select  31:54   0.00% postgres
45325 postgres      1  20    0   183M    37M select  31:52   0.00% postgres
14491 root          4  23    0    94M  6952K sigwai   5:08   0.00% rserver
45139 postgres      1  20    0    31M  3580K select   3:12   0.00% postgres
47069 prosody       1  20    0    24M  8188K select   2:06   0.00% lua52
avery@nixphere:~ % 

12.3 kill

The kill command sends a message to a process identified by it's PID. Although the name sounds dangerous, it can be used for more than killing processes.

avery@nixphere:~ % ps
  PID TT  STAT    TIME COMMAND
32216  0  INs  0:00.05 -tcsh (tcsh)
32272  2  SNs  0:00.20 -tcsh (tcsh)
32666  2  SNC  0:00.00 sleep 1000
32667  2  RN+  0:00.00 ps
avery@nixphere:~ % kill 32666
avery@nixphere:~ % ps
  PID TT  STAT    TIME COMMAND
32216  0  INs  0:00.05 -tcsh (tcsh)
32272  2  SNs  0:00.20 -tcsh (tcsh)
32668  2  RN+  0:00.00 ps
avery@nixphere:~ % 

By default, kill will end a process. If you want to send a different signal, you can use -s $signalNumber . Here are some of the more commonly used signals:


1       HUP (hang up)
2       INT (interrupt)
3       QUIT (quit)
6       ABRT (abort)
9       KILL (non-catchable, non-ignorable kill)
14      ALRM (alarm clock)
15      TERM (software termination signal)

avery@nixphere:~ % ps
  PID TT  STAT    TIME COMMAND
32216  0  INs  0:00.05 -tcsh (tcsh)
32272  2  SNs  0:00.21 -tcsh (tcsh)
32671  2  INC  0:00.00 sleep 1000
32674  2  RN+  0:00.00 ps
avery@nixphere:~ % kill -s 9 32671
avery@nixphere:~ % ps
  PID TT  STAT    TIME COMMAND
32216  0  INs  0:00.05 -tcsh (tcsh)
32272  2  SNs  0:00.21 -tcsh (tcsh)
32675  2  RN+  0:00.00 ps
avery@nixphere:~ % 

12.4 pkill

The pkill command works similarly to kill but it is used to kill processes by their names. Unless you are root, you cannot kill processes belonging to other users. Be careful! pkill will kill all processes that match the name you specify!

avery@nixphere:~ % ps
  PID TT  STAT    TIME COMMAND
32216  0  INs  0:00.05 -tcsh (tcsh)
32272  2  SNs  0:00.22 -tcsh (tcsh)
32723  2  RN+  0:00.00 ps
32682  3  SNs  0:00.03 -tcsh (tcsh)
32721  3  SN+  0:00.01 vi
avery@nixphere:~ % pkill vi
pkill: signalling pid 32471: Operation not permitted
avery@nixphere:~ % ps
  PID TT  STAT    TIME COMMAND
32216  0  INs  0:00.05 -tcsh (tcsh)
32272  2  SNs  0:00.23 -tcsh (tcsh)
32729  2  RN+  0:00.00 ps
32682  3  INs+ 0:00.04 -tcsh (tcsh)
avery@nixphere:~ % ps aux | grep 32471
root     32471  0.0  0.5   22368 10612  0  I+   00:39       0:20.78 vim cli.html
avery@nixphere:~ % 

12.5 jobs

The jobs command lists the jobs currently running in your terminal. It's most useful for listing processes running in the background. IN order to start a job in the background, you can add a & character to the end of your command.

avery@nixphere:~ % sleep 999999 &
[1] 32769
avery@nixphere:~ % vim &
[2] 32770
avery@nixphere:~ % jobs
[1]  - Running                       sleep 999999
[2]  + Suspended (tty output)        vim
avery@nixphere:~ % 

12.6 fg

the fg command is used to place a backgrounded job into the foreground.

very@nixphere:~ % vim &
[2] 32774
avery@nixphere:~ % jobs
[1]  - Running                       sleep 999999
[2]  + Suspended (tty output)        vim
avery@nixphere:~ % fg 2
[ vim here ]

12.7 Control+z

This keyboard shortcut will suspend a process and place it into the background.

avery@nixphere:~ % tail -f /var/log/messages
Aug 11 20:40:47 nixphere sshd[31501]: error: maximum authentication attempts exceeded for invalid user test from 94.75.36.6 port 49796 ssh2 [preauth]
Aug 11 20:40:53 nixphere sshd[31503]: error: maximum authentication attempts exceeded for invalid user test from 94.75.36.6 port 49816 ssh2 [preauth]
Aug 11 20:41:00 nixphere sshd[31507]: error: maximum authentication attempts exceeded for invalid user user from 94.75.36.6 port 49851 ssh2 [preauth]
Aug 11 20:41:05 nixphere sshd[31510]: error: maximum authentication attempts exceeded for invalid user user from 94.75.36.6 port 49866 ssh2 [preauth]
Aug 11 20:41:16 nixphere sshd[31514]: error: maximum authentication attempts exceeded for invalid user ftpuser from 94.75.36.6 port 49898 ssh2 [preauth]
Aug 11 20:41:20 nixphere sshd[31516]: error: maximum authentication attempts exceeded for invalid user ftpuser from 94.75.36.6 port 49911 ssh2 [preauth]
Aug 11 20:41:30 nixphere sshd[31520]: error: maximum authentication attempts exceeded for invalid user test1 from 94.75.36.6 port 49939 ssh2 [preauth]
Aug 11 20:41:34 nixphere sshd[31523]: error: maximum authentication attempts exceeded for invalid user test1 from 94.75.36.6 port 49964 ssh2 [preauth]
Aug 11 20:41:42 nixphere sshd[31527]: error: maximum authentication attempts exceeded for invalid user test2 from 94.75.36.6 port 49995 ssh2 [preauth]
Aug 11 20:41:46 nixphere sshd[31529]: error: maximum authentication attempts exceeded for invalid user test2 from 94.75.36.6 port 50011 ssh2 [preauth]
^Z
Suspended
avery@nixphere:~ % jobs
[1]  + Suspended                     tail -f /var/log/messages

12.8 bg

the bg command will start a suspended process in the background.

avery@nixphere:~ % jobs
[1]  + Suspended                     tail -f /var/log/messages
avery@nixphere:~ % bg 1
[1]    tail -f /var/log/messages &
avery@nixphere:~ % jobs
[1]    Running                       tail -f /var/log/messages
avery@nixphere:~ % 

12.9 Terminal Multiplexing

There is a class of programs called "Terminal Multiplexers". You can think of these as window managers for your terminal. The most usable one is called tmux but some people prefer screen . Multiplexers are useful for when you need to leave a command running in the background for a long period of time, for pair programming over the internet, or for when you just want more shells without starting multiple ssh connections.

12.9.1 tmux

Tmux is very useful for when we want to run jobs in the background. It also supports multiple windows, panes, and sessions.

Starting a new session:

avery@nixphere:~ % tmux
[ tmux begins here ] 

as you can see, tmux is just a fancy wrapper for your terminal. Let's make some splits. To split the vertically, use Control + b %. To split horizontally, use Control + b ". To switch between panes, you can use Control + b and any arrow key in the direction you want to travel. To cycle through panes, you can use Control + b o.

These keyboard combinations might seem confusing at first but they become easier to remember when we think of them as key-chords. First you press the keyboard shortcut, then the character after it. Control + b combination is a prefix that sends a command to tmux.

Over quota?  "du -sh * | sort -h " will give you a sorted list of your
directory sizes.
-- David Scheidt 
avery@nixphere:~ % 







────────────────────────────────────────┬───────────────────────────────────────
er to do 'umount /cdrom' before         │Want colour in your directory listings?
removing the CD-ROM (it will usually not│  Use "ls -G".  "ls -F" is also useful,
be possible to remove the CD-ROM        │and they can be combined as "ls -FG".
without doing this.)                    │avery@nixphere:~ % 
                                        │
Note: This tip may not work in all confi│
gurations.                              │
avery@nixphere:~ %                      │
                                        │
                                        │
                                        │
[1] 0:tcsh*                                       "nixphere.org" 00:19 13-Aug-21

Tmux also supports multiple windows. To create a new window, you can use Control + b c. You can use this command as many times as you'd like. In order to switch between windows, you can use the Control + b prefix then type the number of the window you want to switch to. The bottom status line will list your windows. Here is an example:

[1] 0:tcsh  1:R* 2:vim-                           "nixphere.org" 00:35 13-Aug-21

The most useful part of tmux is yet to come. With tmux, we can attach and detach sessions at will. With this ability, we can start our long running process in a tmux session, detach tmux, log out, come back a week later, then attach the session and be right back where we left off. The Control + b d shortcut will detach a session. To list sessions, we can use the tmux ls command. In order to re-open the previous session, we can use the tmux attach command. tmux attach -t $session number can be used to re-open a specific previous session. All tmux sessions are automatically numbered.

avery@nixphere:~ % tmux ls
0: 1 windows (created Fri Aug 13 00:12:47 2021) (attached)
5: 3 windows (created Fri Aug 13 00:32:21 2021)
avery@nixphere:~ % tmux attach -t 5
[ tmux begins here ] 

here is a short demo video showing attaching a session, splits, and how tmux deals with terminals of varying sizes.

Written by Avery. This work is licensed under a CC BY-SA 4.0 License