Subsections of Linux Commands

Filesystem

df & du

  • df - report file system disk space usage
  • du - estimate file space usage
$ df -h  /dev/mmcblk0*   # h human-readable
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
/dev/mmcblk0p1  253M   99M  154M  39% /boot/firmware
/dev/mmcblk0p2   15G  3.5G   11G  26% /

$ du -sh /mnt/sgt2t/     # s display only a total for each argument
633G    /mnt/sgt2t/

Files and directory's

  • “On a UNIX system, everything is a file; if something is not a file, it is a process.”
  • A directory is just a file containing names of other files

pwd

Lists path of the current working directory (folder)

$ pwd
/home/ubuntu

cd

Change directory command is used to change the current working directory

$ cd /var/log/journal/   # cd to /var/log
$ pwd
/var/log/journal/
$ cd ..                   # cd one path up
$ cd ../..                # cd two paths up, etc
$ cd                      # cd to home dir
$ cd -                    # cd to previous dir

ls

Lists the contents of a directory

$ ls /var/log/
$ ls -a /var/log/         # do not ignore entries starting with . (hidden files)
$ ls -l /var/log/         # use a long listing format
$ ls -alh /var/log/       # human-readable with -l and -s, print sizes like 1K 234M 2G etc.
$ ls -ltr /var/log/       # t (sort by modification time, newest first) r (reverse order while sorting)

cat

Short for concatenate, it is used to list the contents of a file on the standard output (stdout)

$ cat metasyntactic-vars.txt
foo
bar
buz
qux

cp

Copy Files

# copy source destination
cp metasyntactic-vars.txt metasyntactic-vars

mv

Move or Rename Files

# mv source destination
$ mv metasyntactic-vars metasyntactic-variables   # rename file
$ mv metasyntactic-vars.txt /tmp/                 # move file to /tmp dir
$ ls /tmp
metasyntactic-vars.txt
$ mv /tmp/metasyntactic-vars.txt .                # move file to current location

mkdir & rmdir

Make a new directory or remove dir

mkdir /tmp/foo                     # make a new dir foo in /tmp
mkdir -p /tmp/bar/foo              # make a new dir foo in /tmp/bar and create parent directories as needed
mkdir -p /tmp/test/{foo,bar,buz}   # make dirs foo bar buz in /tmp/test/ and create parent directories as needed
rmdir /tmp/foo                     # removes a non-empty directory

rm

Remove files or directories.

rm /tmp/foo                 # removes a non-empty directory
rm -r /tmp/bar/foo          # removes foo directory
rm metasyntactic-vars.txt   # removes metasyntactic-vars.txt file

file

Determine file type

$ file var/log/dmesg
var/log/dmesg: UTF-8 Unicode text, with very long lines
$ file /bin/bash
/bin/bash: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=2ee5bcc1055c34df331b2493918ee6bb7119df1c, for GNU/Linux 3.7.0, stripped
$ file sbin
sbin: symbolic link to usr/sbin

locate

locate a file

locate syslog

find

Search for a file.

find . -name testfile.txt           # find a file called testfile.txt in current and sub-directories.
find /home/ubuntu -name *.txt       # find all .txt files in the /home/ubuntu.
find /var/ -name sys* 2>/dev/null  # find everything that starts with sys in /var/ and sub-directories
                                   # and redirect errors to /dev/null
# https://www.tecmint.com/35-practical-examples-of-linux-find-command/

Text Files

Working with text files

head & tail

  • head - output the first part of files
  • tail - output the last part of files
head /var/log/syslog             # Print the first 10 lines to standard output
head -n 1 /var/log/syslog        # print first line to stdout
tail -20  /var/log/syslog        # print last 30 lines from syslog
tail -20f  /var/log/syslog       # print last 30 lines from syslog and follow

diff

compare files line by line

diff  /tmp/test/foo/metasyntactic-vars /tmp/test/bar/metasyntactic-vars
diff -c /tmp/test/foo/metasyntactic-vars /tmp/test/bar/metasyntactic-vars  # context
diff -u /tmp/test/foo/metasyntactic-vars /tmp/test/bar/metasyntactic-vars  # unified

grep

Stands for Global Regular Expression Print. grep, egrep, fgrep, rgrep - print lines that match patterns

$ grep foo /tmp/test/foo/metasyntactic-vars     # search foo in /tmp/test/foo/metasyntactic-vars file
foo
$ grep -v foo /tmp/test/foo/metasyntactic-vars  # list everything but foo in /tmp/test/foo/metasyntactic-vars file
bar
buz
quz
grep -i FOO /tmp/test/foo/metasyntactic-vars    # same as above, i sands for ignore case
$ grep -r buz /tmp/test/                        # search recursively for buz in all files from /tmp/test dir
/tmp/test/buz/metasyntactic-vars:buz
/tmp/test/bar/metasyntactic-vars:buz
/tmp/test/foo/metasyntactic-vars:buz
$ egrep "foo|bar" /tmp/test/foo/metasyntactic-vars  # search for both foo and bar in file
foo
bar

tr

stands for translate

# replace spaces with new lines
echo 'alpha beta gamma delta alpha beta gamma delta' | tr " " "\n"

sort & uniq

sort - sort lines of text files uniq - report or omit repeated lines (works with sorted lines)

# sorts the line
echo 'alpha beta gamma delta alpha beta gamma delta' | tr " " "\n" | sort
# shows uniq lines
echo 'alpha beta gamma delta alpha beta gamma delta' | tr " " "\n" | sort | uniq
# prefix lines by the number of occurrences
echo 'alpha beta gamma delta beta gamma delta' | tr " " "\n" | sort | uniq -c

awk

awk (gawk) - pattern scanning and processing language

$ echo 'a|b|c' | awk -F "|" '{print $2}'
b
$ echo 'a b c' | awk '{print $2}'
b
cat /etc/passwd | awk -F ":" '{printf "%-25s %s\n" , $1, $7}'  # print user and shell
cat /etc/passwd | awk -F ":" '{printf "%-25s %s\n" , $1, $NF}' # print user and shell $NF is the last column

sed

sed - stream editor for filtering and transforming text

$ echo 'a b b c' | sed 's/b/x/'    # substitutes first encountered `b`
a x b c
$ echo 'a b b c' | sed 's/b/x/g'   # substitutes globally all `b`
a x x c

Networking

ip & ifconfig

ip - show / manipulate routing, network devices, interfaces and tunnels ifconfig - configure a network interface

ip address show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether dc:a6:32:ac:da:5b brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.10/24 brd 192.168.100.255 scope global dynamic eth0
       valid_lft 170906sec preferred_lft 170906sec
    inet6 fe80::dea6:32ff:feac:da5b/64 scope link
       valid_lft forever preferred_lft forever

ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.10  netmask 255.255.255.0  broadcast 192.168.100.255
        inet6 fe80::dea6:32ff:feac:da5b  prefixlen 64  scopeid 0x20<link>
        ether dc:a6:32:ac:da:5b  txqueuelen 1000  (Ethernet)
        RX packets 2508154727  bytes 876813076579 (876.8 GB)
        RX errors 185085  dropped 71761560  overruns 0  frame 0
        TX packets 3770704443  bytes 5244870710506 (5.2 TB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ping

The ping utility uses the ICMP protocol’s mandatory ECHO_REQUEST datagram to elicit an ICMP ECHO_RESPONSE from a host or gateway

# ping find out if a host is alive
# this command causes false negative if ICMP protocol is blocked by the host
PING google.ro (172.217.19.99): 56 data bytes
64 bytes from 172.217.19.99: icmp_seq=0 ttl=118 time=32.413 ms
64 bytes from 172.217.19.99: icmp_seq=1 ttl=118 time=32.734 ms
64 bytes from 172.217.19.99: icmp_seq=2 ttl=118 time=31.968 ms

dig

dig (domain information groper) is a flexible tool for interrogating DNS name servers.

# print the destination IP for the A Record
$ dig +short google.ro
172.217.18.67
# print answer section only
$ dig +noall +answer google.ro
google.ro.        138    IN    A    172.217.19.99

nslookup

Nslookup is a program to query Internet domain name servers.

$ nslookup google.ro
Server:        192.168.100.1
Address:    192.168.100.1#53

Non-authoritative answer:
Name:    google.ro
Address: 172.217.18.67

traceroute

traceroute – print the route packets take to network host

$ traceroute google.ro
traceroute to google.ro (172.217.19.99), 64 hops max, 52 byte packets
 1  192.168.100.1 (192.168.100.1)  4.364 ms  1.155 ms  0.993 ms
 2  109.98.30.1 (109.98.30.1)  3.786 ms  3.916 ms  3.441 ms
13  bud02s27-in-f3.1e100.net (172.217.19.99)  32.314 ms  33.097 ms  31.379 ms

telnet

User interface to the TELNET protocol. Useful to test remote open ports.

# port is open but refused to accept telnet connection
$ telnet 192.168.100.128 8080
Trying 192.168.100.128...
telnet: connect to address 192.168.100.128: Connection refused
telnet: Unable to connect to remote host
# port is not open
$ telnet 192.168.1.128 8081
Trying 192.168.1.128...
^C

Others

hostname  # print hostname or fqdn

File transfer

Transfer files between servers

scp

scp — OpenSSH secure file copy

# cp remote file to work dir
scp ubuntu@raspberrypi-live:~/work/readme.md ~/work
# cp file to remote dir
scp ~/work/readme.md ubuntu@raspberrypi-live:~/work/

rsync

rsync - a fast, versatile, remote (and local) file-copying tool

It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination.

# sync current dir to remote dir
rsync -avz . ubuntu@raspberrypi-live:~/work
# sync remote dir to current dir
rsync -avz ubuntu@raspberrypi-live:~/work .

Processes

List processes

top & htop

  • top provides a dynamic real-time view of a running system. More details in man top.
  • htop It is similar to top, but allows you to scroll vertically and horizontally, so you can see all the processes running on the system, along with their full command lines, as well as viewing them as a process tree, selecting multiple processes and acting on them all at once. More details in man htop.
top
top -U ubuntu  # top user processes
htop

ps

ps displays information about a selection of the active processes.

ps -ef                                       # (-e) list all processes and (-f) do full-format listing
ps axjf        #   ps -ejH                   # To print a process tree
ps axms        #   ps -eLf                   # To get info about threads
ps -eo euser,ruser,suser,fuser,f,comm,label  # To get security info
ps axZ         #   ps -eM                    # To get security info
ps -u ubuntu                                 # list only ubuntu user processes
ps -eo pid,nice,user,args --sort nice        # user-defined format
ps -eo '%p %n %u %a' --sort nice             # same as above but with aix format descriptors

pstree

display a tree of processes

pstree
pstree -a
pstree -a -C age

pidof

Find the process ID of a running program.

$ pidof dockerd         # find the pid of a program
2000
$ pidof -x ./script.sh  # find the pid of a script
38908

kill, pkill & killall

  • kill - send a signal to a process
  • pgrep, pkill - look up or signal processes based on name and other attributes
kill [PID]                #
kill -9 [PID]             # SIGKILL aka force kill
kill -15 [PID]            # SIGTERM aka  nice kill
kill -l                   # list all signals

pgrep -u root dockerd     # Find the process ID of the dockerd daemon
pkill -HUP syslogd        # Make syslog reread its configuration file
renice +4 $(pgrep chrome) # Make all chrome processes run nicer

Scheduling

A kernel scheduler is a unit of the kernel that determines the most suitable process out of all runnable processes to execute next. The scheduler forms the core of multitasking in Linux. It ranks processes based on the most deserving as well as the need for CPU time

One important characteristic of Linux is dynamic priority-based scheduling, which allows the nice value of processes to be changed (increased or decreased) depending on your needs, as we’ll see later on.

Niceness values range from -20 (most favorable to the process) to 19 (least favorable to the process).

nice & renice

  • nice - run a program with modified scheduling priority
  • renice - alter the priority of running processes
# -e Select all processes.
# -o user-defined format
nice -5 tar -czf backup.tar.gz /home/ubuntu   # nice -PRIORITY COMMAND
ps -eo pid,nice,user,args --sort nice         # determine the niceness of a command
$ ps -eo pid,nice,user,args | grep dockerd
   2000   0 root     /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
$ renice -n -2  -u dockerd
# or
$ renice -n -2  -p 2000

nohup

Run a command immune to hangups, with output to a non-tty

nohup backup-database.sh    # execute a long running command
nohup backup-database.sh &  # execute a long running command and continue to work in terminal

Job Controlling commands

# create a dummy script that appends a date to a temp file
$ printf "%s\n%s\n"  '#!/bin/bash' 'while true; do echo $(date --rfc-3339=seconds) >>/tmp/running.test ; sleep 1 ; done' > script.sh
$ chmod +x script.sh            # add execute flag to script
$ ./script.sh                   # run script
$ ctrl+z                        # hit control + z
^Z
[1]+  Stopped                 ./script.sh
$ jobs                          # list jobs
[1]+  Stopped                 ./script.sh
$ bg %1                         # run job 1 in background
$ tail -10f /tmp/running.test   # check if script is running in background
2020-07-02 17:26:54+00:00       # <>
2020-07-02 17:27:10+00:00       # time when job was stopped with ctrl+z
2020-07-02 17:27:11+00:00
2020-07-02 17:27:12+00:00
jobs
[1]+  Running                 ./script.sh &
kill %1                        # kill job with id 1
$ jobs
[1]+  Terminated              ./script.sh
$ ./script.sh
^Z
[1]+  Stopped                 ./script.sh
stop %1                       # stops a bg process

References:

Others

tar

An archive utility

tar -cvf test.tar /tmp/test/     # Create Verbosely Filename
tar -cvzf test.tar.gz /tmp/test/ # Create Verbosely gzip Filename
# instructs tar to change dir to /tmp/test and start create archive from there
tar --strip-components=1 -cvzf test.tar.gz -C /tmp/test/ .

tar -tvf test.tar.gz             # List Verbosely archive content

tar -xvf test.tar                # Extract Verbosely Archive
tar -xvf test.tar.gz             # Extract Verbosely gzip Archive
# extract a single file from an archive
tar --extract --file=test.tar.gz tmp/test/foo/metasyntactic-vars

lsof

List open files. Useful to see what application is running on some port.

$ lsof -i                # show open files in this case TCP/UDP ports (might require root)
$ sudo lsof -i :8080     # find what application is running and listen for connections on port 8080
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 2282 root    4u  IPv6  53248      0t0  TCP *:http-alt (LISTEN)
$ ps -ef | grep 2282     # find out more about the PID
root        2282    2000  0 Jun23 ?        00:00:01 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8080 -container-ip 172.17.0.2 -container-port 8080

read

read - reads from stdin into variables.

$ echo "alpha beta gamma delta" | while read A B C D ; do echo $D $C $B $A ; done
delta gamma beta alpha
$ echo "alpha beta gamma delta" | while read first_word rest_of_the_line; do echo $first_word ; done
alpha
$ echo "alpha beta gamma delta" | while read first_word rest_of_the_line; do echo $rest_of_the_line ; done
beta gamma delta

TODO

ln                      #
nslookup google.ro      # query Internet name servers
ping                    # ping a remote host, ping uses ICMP type 8 (echo message) and type 0 (echo reply message)
                        # some host block ICMP giving a false-positive of unreachable host

crontab                 #
regex                   #
xargs                   #
eval                    #
trap                    # trap

Other