Why UNIX for Data Science?

• Why so many commands? Isn't GUI nicer, faster and overall better?

• The answer is yes: the GUI is nicer and faster at discovering various options but when it comes to pure execution, CLI tends to be unmatched in speed

• In other words, by using a mouse you don't build muscle memory

Why UNIX for Data Science?

• Why so many commands? Isn't GUI nicer, faster and overall better?

• The answer is yes: the GUI is nicer and faster at discovering various options but when it comes to pure execution, CLI tends to be unmatched in speed

• In other words, by using a mouse you don't build muscle memory

• Great tools for your toolbox

• Parallel processing

A tale of grep

Suppose you would like to find a specific like (say "guide to the galaxy") you know is saved in some file on the disk.

How would you go about doing that?

A tale of grep

Suppose you would like to find a specific like (say "guide to the galaxy") you know is saved in some file on the disk.

How would you go about doing that?

Well, turns out grep can be of help!

Up until now we used it in the following way

$ grep [regex] [file]

A tale of grep

Suppose you would like to find a specific like (say "guide to the galaxy") you know is saved in some file on the disk.

How would you go about doing that?

Well, turns out grep can be of help!

Up until now we used it in the following way

$ grep [regex] [file]

But it can also be applied recursively on directories and their content.

grep with recursion

grep -R

• read all files under each directory, recursively, and follow symbolic links
• starts from the current working directory by default

grep with recursion

grep -R

• read all files under each directory, recursively, and follow symbolic links
• starts from the current working directory by default

$ tree
.
├── a
│   ├── file1.txt
│   └── file2.txt
├── b
│   └── file40.txt
├── c
│   └── file3.txt
├── d
└── g
    └── book.txt
5 directories, 5 files

$ grep -Ri 'guide to the galaxy'
g/book.txt:The Hitchiker's Guide to the Galaxy

grep's Extended Regular Expressions

• grep uses so called "basic regular expressions" (BRE) by default.

grep's Extended Regular Expressions

• grep uses so called "basic regular expressions" (BRE) by default.

• What we normally consider "regular expressions" are actually "extended regular expressions" (ERE)

grep's Extended Regular Expressions

• grep uses so called "basic regular expressions" (BRE) by default.

• What we normally consider "regular expressions" are actually "extended regular expressions" (ERE)

• The main difference is in backslashing the meta characters:

  • BRE: \?, \+, \{, \}
  • ERE: ?, +, {, }

EREs can be turned on in grep by passing the -E parameter.

grep's Extended Regular Expressions

• grep uses so called "basic regular expressions" (BRE) by default.

• What we normally consider "regular expressions" are actually "extended regular expressions" (ERE)

• The main difference is in backslashing the meta characters:

  • BRE: \?, \+, \{, \}
  • ERE: ?, +, {, }

EREs can be turned on in grep by passing the -E parameter.

Sample: find all Uniba login IDs in the current directory:

grep's Extended Regular Expressions

• grep uses so called "basic regular expressions" (BRE) by default.

• What we normally consider "regular expressions" are actually "extended regular expressions" (ERE)

• The main difference is in backslashing the meta characters:

  • BRE: \?, \+, \{, \}
  • ERE: ?, +, {, }

EREs can be turned on in grep by passing the -E parameter.

Sample: find all Uniba login IDs in the current directory:

$ grep -Rn -E '[a-z]+[0-9]{1,3}'
l/access.log:11:Unauthorized access attmept from login ID novak123
l/access.log:18:Successfuly authorized roy47
z/login.py:4:log_in(username='rob5')

grep's Extended Regular Expressions

• grep uses so called "basic regular expressions" (BRE) by default.

• What we normally consider "regular expressions" are actually "extended regular expressions" (ERE)

• The main difference is in backslashing the meta characters:

  • BRE: \?, \+, \{, \}
  • ERE: ?, +, {, }

EREs can be turned on in grep by passing the -E parameter.

Sample: find all Uniba login IDs in the current directory:

$ grep -Rn -E '[a-z]+[0-9]{1,3}'
l/access.log:11:Unauthorized access attmept from login ID novak123
l/access.log:18:Successfuly authorized roy47
z/login.py:4:log_in(username='rob5')

But what if we only wanted to search in say .py files?

find

• A command for locating (finding) files and directories
• It looks through all the subdirectories recursively
• If no starting-point is specified it starts from .

$ tree
.
├── a
│   ├── file1.txt
│   └── file2.txt
├── b
│   └── file40.txt
├── c
│   └── file3.txt
└── d
4 directories, 4 files

$ find
.
./d
./c
./c/file3.txt
./b
./b/file40.txt
./a
./a/file2.txt
./a/file1.txt
$ find .
.
./d
./c
./c/file3.txt
./b
./b/file40.txt
./a
./a/file2.txt
./a/file1.txt

find: looking up filenames

Can be done with the -name option / flag

$ find .
.
./d
./c
./c/file3.txt
./b
./b/file40.txt
./a
./a/file2.txt
./a/file1.txt

$ find . -name "*.txt"
./c/file3.txt
./b/file40.txt
./a/file2.txt
./a/file1.txt

$ find . -name "file?.txt"
./c/file3.txt
./a/file2.txt
./a/file1.txt

$ find . -name 'file[2-4].txt'
./c/file3.txt
./a/file2.txt

find: looking up paths

For matching full paths, -path is a good choice

$ find .
.
./d
./c
./c/file3.txt
./b
./b/file40.txt
./a
./a/file2.txt
./a/file1.txt

$ find . -path '*a*'
./a
./a/file2.txt
./a/file1.txt

$ find . -path './?'
./d
./c
./b
./a

$ find . -path '*c/file[0-9].txt'
./c/file3.txt

$ find . -path '*c/file[0-9]'

find: looking up via regex

If wildcards do not suffice, we can also utilize the "full power of regexes"

$ find .
.
./d
./c
./c/file3.txt
./b
./b/file40.txt
./a
./a/file2.txt
./a/file1.txt

$ find . -regex '.*file.*'
./c/file3.txt
./b/file40.txt
./a/file2.txt
./a/file1.txt

$ find . -regex '.*[bc]/file.*'
./c/file3.txt
./b/file40.txt

$ find . -regex 'file'

$ find . -regex '[bc]/file.*'

find: looking up via attributes

Files and directories have various attributes find can take a look at:

• type

  • directory, file, symlink, ...

• timestamps

  • last file change (-ctime)
  • last access (-atime)
  • last modification (-mtime)

• file size

• owner and group

• permissions

find: looking up by file/dir type

$ find .
.
./d
./c
./c/file3.txt
./b
./b/file40.txt
./a
./a/file2.txt
./a/file1.txt

$ find . -type d
.
./d
./c
./b
./a

$ find . -type f
./c/file3.txt
./b/file40.txt
./a/file2.txt
./a/file1.txt

find: looking up by timestamps

$ ls -al
drwxrwxr-x 7 mrshu mrshu 4096 Nov  9 10:34 .
drwxrwxr-x 2 mrshu mrshu 4096 Nov  7 11:39 a
drwxrwx--x 2 mrshu mrshu 4096 Nov  7 11:40 b
drwxrwxr-x 2 mrshu mrshu 4096 Nov  7 11:57 c
drwxrwxr-x 2 mrshu mrshu 4096 Nov  7 12:36 d
drwxrwxr-x 2 mrshu mrshu 4096 Nov  9 10:34 e

$ date
Mon 09 Nov 2020 10:55:23 AM UTC

$ find . -mmin 22
.
./e

$ find . -mtime 1
./a
./d
./b
./c

find: looking up by size

$ find .
.
./d
./c
./c/file3.txt
./b
./b/file40.txt
./a
./a/file2.txt
./a/file1.txt

$ find . -empty
./d
./c/file3.txt
./a/file1.txt

$ find . -size 0
./c/file3.txt
./a/file1.txt

find: a note on [n]

By default, [n] matches the exact value (of time/date or size)

This behaviour can be altered via the + and - prefixes

• +[n]

  • matches all values larger than [n]

• -[n]

  • matches all values smaller than [n]

• [n]

  • matches exactly [n]

$ find . -mtime -3
.
./a
./d
./b
./c
./e

$ find /boot -size +15M
/boot/initrd.img-5.4.0-47-generic
/boot/initrd.img-5.4.0-52-generic
/boot/initrd.img-5.4.0-51-generic
$ ls -hs /boot/initrd.img-5.4.0-47-generic
78M /boot/initrd.img-5.4.0-47-generic

find: looking up by user/group

find [path] -user [user]

• only show files and directories owned by [user]

find [path] -group [group]

• only show files and directories that belong to group [group]

$ ls -l /etc
[ ... 160 lines omitted ... ]
drwxr-xr-x 2 root root       4096 Sep 19 19:52 sensors.d
-rw-r--r-- 1 root root      14464 Feb 16  2020 services
-rw-r----- 1 root shadow     1463 Sep 19 20:15 shadow
-rw-r----- 1 root shadow     1595 Sep 19 20:14 shadow-
-rw-r--r-- 1 root root        146 Jul 31 16:29 shells
drwxr-xr-x 2 root root       4096 Jul 31 16:28 skel
[ ... 32 lines omitted ... ]
$ find /etc/ -group shadow
/etc/shadow-
/etc/gshadow
/etc/shadow
/etc/gshadow-

find: looking up by permissions

find [path] -readable

• only show files that are readable

find [path] -writable

• only show files that are writable

find [path] -executable

• only show files that are executable

$ find .
.
./d
./c
./c/file3.txt
./b
./b/file40.txt
./a
./a/file2.txt
./a/file1.txt

find . -executable
.
./d
./c
./b
./a

find: looking up by permissions II

find [path] -perm [mode]

-[mode] can be specified in octal or symbolic format

• [mode] can have various prefixes:
  • [mode]: exactly [mode] permissions are set
  • -[mode]: at least [mode] permissions are set
  • /[mode]: some [mode] permissions are set

$ ls -al
total 24
drwxrwxr-x 6 mrshu mrshu 4096 Nov  7 12:36 .
drwxr-xr-x 7 mrshu mrshu 4096 Nov  9 10:16 ..
drwxrwxr-x 2 mrshu mrshu 4096 Nov  7 11:39 a
drwxrwx--x 2 mrshu mrshu 4096 Nov  7 11:40 b
drwxrwxr-x 2 mrshu mrshu 4096 Nov  7 11:57 c
drwxrwxr-x 2 mrshu mrshu 4096 Nov  7 12:36 d
$ find . -perm 771
./b
$ find . -perm -774
.
./a
./d
./c

find: combining search patterns

The search pattern on name and attribute level can be easily combined together.

For example:

• find all empty .txt files

$ file . -name "*.txt" -empty

• find all files modified in the last 20 minutes, whose filename contains "image"

$ file . -name "*image*" -mmin -20

find: actions on matches

-delete

• delete all matched files

-exec [command] \;

• run command [command] for each matched file/directory
• string {} is replaced with the matched file/directory

-ok [command] \;

• same thing as -exec but asks for user confirmation before running the command

$ cat c/file3.txt
$ cat b/file40.txt
This is 40
$ cat a/file2.txt
This is us
$ cat a/file1.txt

$ find . -name "*.txt" -exec echo {} \;
./c/file3.txt
./b/file40.txt
./a/file2.txt
./a/file1.txt
$ find . -name "*.txt" -exec cat {} \;
This is 40
This is us

xargs

• Allows us to "parametrize" the commands we run by passing input via pipe

• Processes standard input line-by-line and "applies" a command on each line

xargs [command]

• command can be any command we would like to run on each line

• -I {} will replace {} in the [command] by the input

$ find . -type f | xargs -I{} echo "File: {}"
File: ./c/file3.txt
File: ./b/file40.txt
File: ./a/file2.txt
File: ./a/file1.txt

xargs

• Allows us to "parametrize" the commands we run by passing input via pipe

• Processes standard input line-by-line and "applies" a command on each line

xargs [command]

• command can be any command we would like to run on each line

• -I {} will replace {} in the [command] by the input

$ find . -type f | xargs -I{} echo "File: {}"
File: ./c/file3.txt
File: ./b/file40.txt
File: ./a/file2.txt
File: ./a/file1.txt

This can be easily used as a clearer alternative to -delete or -exec

$ find . -type f -empty | xargs -I{} rm {}

xargs II

xargs [command]

• -t

  • print each command prior to execution

• -p

  • similar to -ok in find
  • ASKS For confirmation before executing a command

• -P [max-procs]

  • run the commands in parallel, on at most [max-procs] processes

Ask before removing each empty file:

$ find . -type f -empty | xargs -p -I{} rm {} 
rm ./c/file3.txt ?...y
rm ./a/file1.txt ?...y

Back to the grep tale

Task: find all Uniba IDs in the *.py files in the current directory (recursively)

Solution:

$ find . -name '*.py' -type f | xargs grep -n -E '[a-z]+[0-9]{1,3}'
z/login.py:4:log_in(username='rob5')

time

• Let's you "time" (compute the time needed for) the execution of a specific command

• An internal bash command but a standalone program also exists

• Very nice for benchmarking competing approaches to solving the same problem

time

• Let's you "time" (compute the time needed for) the execution of a specific command

• An internal bash command but a standalone program also exists

• Very nice for benchmarking competing approaches to solving the same problem

Suppose we'd like to benchmark the following two commands:

find ./foo -type f -name "*.txt" -exec rm {} \; 
find ./foo -type f -name "*.txt" | xargs rm

time

• Let's you "time" (compute the time needed for) the execution of a specific command

• An internal bash command but a standalone program also exists

• Very nice for benchmarking competing approaches to solving the same problem

Suppose we'd like to benchmark the following two commands:

find ./foo -type f -name "*.txt" -exec rm {} \; 
find ./foo -type f -name "*.txt" | xargs rm

On a folder with 1000 files in it, here are the results:

time find ./foo -type f -name "*.txt" -exec rm {} \;
0.35s user 0.11s system 99% cpu 0.467 total
time find ./foo -type f -name "*.txt" | xargs rm
0.00s user 0.01s system 75% cpu 0.016 total

As we can see, the xargs approach seems to be a bit faster (various benchmarks tend to agree)

timeing the parallel xargs

Let's use time to demonstrate the difference that the -P in xargs can have.

The program we'll test this on is very simple -- just sleep (wait) for a bit (like an extensive computation would).

Sleep for 1, 2, 3, 4 and 5 seconds serially (about 15 sec in total):

$ time echo 1 2 3 4 5 | tr ' ' '\n' | xargs -I{} sleep {}
real    0m15.017s
user    0m0.007s
sys     0m0.012s

Sleep for 1, 2, 3, 4 and 5 seconds in parallel (about 5 sec in total):

$ echo 1 2 3 4 5 | tr ' ' '\n' | xargs -P 5 -I{} sleep {}
real    0m5.012s
user    0m0.007s
sys     0m0.014s