{ "cells": [ { "cell_type": "markdown", "id": "ebee68e1-7767-4faf-a46b-b6a1e8b8069d", "metadata": { "hide_input": false, "slideshow": { "slide_type": "slide" } }, "source": [ "% SLIDE TITLE\n", "# SLS Lecture 2 : Shell Part II : Programming, Files and Directories" ] }, { "cell_type": "markdown", "id": "2be12114-6cbd-4979-91a7-e7b7a4094865", "metadata": { "hide_input": false, "tags": [] }, "source": [ "% SLIDE BODY\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", "

\n", " Getting our hands dirty.
\n", " As powerful as UNIX and the shell are, they are also old and dirty... and have had plenty of time to accumulate some Barnacles. \n", "

\n", "
" ] }, { "cell_type": "markdown", "id": "fc59c26c-f7f0-4968-afdd-fdb66fefd88d", "metadata": { "slideshow": { "slide_type": "skip" }, "tags": [ "remove-cell" ] }, "source": [ "
" ] }, { "cell_type": "markdown", "id": "6b63f8b5-f877-4e33-bbd7-be4c0147eb4a", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "% SLIDE TITLE\n", "## Our goals:" ] }, { "cell_type": "markdown", "id": "3ef1e7fc-b46a-42a9-a485-499beff9788c", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "% SLIDE BODY\n", "1. Basic proficiency in \n", " - using the shell as a programming language \n", " - core to automating your UNIX life\n", " - interactively : one liners\n", " - scripts : small and large programs \n", " - navigating and working with the file system\n", " - running and managing running programs\n", " - IO redirection and pipes\n", " - processes\n", " - understand basics of UNIX credentials\n", " - user and groups names and ids, file permissions, and who \"root\" is" ] }, { "cell_type": "markdown", "id": "36539305", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "2. Enable you to learn more\n", " - know enough so that you can experiment\n", " - know where you can look for more info\n", " - know enough so you can appreciate and learn from others\n", " - idioms, tricks, and hacks\n", " - scripts " ] }, { "cell_type": "markdown", "id": "b1906c7f", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "3. Understand how things work so that you have the context for\n", " - working efficiently with large bodies of ascii files \n", " - at the command line\n", " - and in editors like `vim` and `emacs`\n", " - `make`\n", " - `git`" ] }, { "cell_type": "markdown", "id": "1ac7a230-ae27-46ab-89b5-3e6a41b8795e", "metadata": { "slideshow": { "slide_type": "skip" }, "tags": [ "remove-cell" ] }, "source": [ "
" ] }, { "cell_type": "markdown", "id": "f051f016-5315-48d5-bb2f-e5895df7b80f", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "## Anatomy of shell command lines \n", "\n", "**How to have a conversation with the shell**\n", "\n", "- \"prompt\" indicates the shell is ready \n", "- Think of command lines like \"sentences\"\n", " - sequences of \"word\" -- tokens\n", " - various characters can be used to split word -- easiest are \"white space\" characters \n", " - quotes can be use to suppress splitting \n", " - newline : 0xa -- '\\n' terminates our sentences -- like our period\n", " - command processing starts\n", "- simple command lines \n", " - first word is command\n", " - rest are arguments to the command\n", "- \"syntax\" will let us compose more complicated \"sentences\"\n", " - combine multiple commands into a single command line" ] }, { "cell_type": "markdown", "id": "864e06dd-474e-4211-90e1-ce8bbb6779b1", "metadata": { "slideshow": { "slide_type": "notes" }, "tags": [] }, "source": [ "**NOTES**\n", "\n", "See book for details \n", "- some \"special\" characters (\"meta-characters\") have special meaning \n", " - | & ; ( ) < > newline \n", " - note these will also cause things to be split into words\n", " - eg things on the right will be treated as a new word \n", " - sometimes they will also terminate a simple command (thing on the right will be a new command)\n", "- some special words (\"reserved\") have special meaning depending on where they occur on the line\n" ] }, { "cell_type": "markdown", "id": "b93f1ab1-64bc-4e16-aa26-e15e820d1736", "metadata": { "slideshow": { "slide_type": "skip" }, "tags": [ "remove-cell" ] }, "source": [ "
" ] }, { "cell_type": "markdown", "id": "7aa7cdbb", "metadata": { "slideshow": { "slide_type": "skip" }, "tags": [] }, "source": [ "## The Shell as a Programming Language\n", "\n", "**The Glue that holds the UNIX Universe together** \n", "\n", "Basic Syntax : Notes\n", "- comments\n", "- echo and printf\n", "- simple variable syntax and usage\n", " - quoting to control splitting and expansion\n", " - many more fancy variable expansions and special variables : `help variables` but plenty more\n", "- basic programming constructs\n", " - for lists of words \n", " - commands and exit codes\n", " - (())\n", " - [[]]\n", " - [ ]\n", "- interactive vs scripts\n", " - history and arrow keys\n", " - scripts\n", " - ascii file with lines of shell commands\n", " - mark as executable and put in path or name explicitly" ] }, { "cell_type": "markdown", "id": "26569462-070e-4266-ae10-92a894e35d51", "metadata": { "slideshow": { "slide_type": "skip" }, "tags": [] }, "source": [ "
" ] }, { "cell_type": "markdown", "id": "37afdd02-fa57-4ea9-b377-f35ea26c9831", "metadata": { "slideshow": { "slide_type": "slide" }, "tags": [] }, "source": [ "## Shell Programming by example" ] }, { "cell_type": "markdown", "id": "4e0534bd-26f0-4c68-b92c-056675dcccbe", "metadata": {}, "source": [ "**USE TERMINAL and ILLUSTRATE**\n", "- Much of the built in capabilities of the shell is to provide us a programming interface\n", " - comments\n", " - set and use variables\n", " - loops: repeat commands\n", " - support for basic math\n", " - exit status -- a way to support conditional constructs\n", " - (( stmt )) : can do math statement and return an exit status\n", " - [[ test ]] : can do a bunch of tests and return an exit status\n", " - the exit status of external commands" ] }, { "cell_type": "markdown", "id": "c5f16fa8", "metadata": { "slideshow": { "slide_type": "notes" }, "tags": [] }, "source": [ "## Shell Programming \n", "
\n", " \n", "``` bash\n", "# Comments\n", "# a word that starts with # and everything else till end end of the line is ignored\n", "\n", "# echo - basic built in for outputting\n", "echo hello\n", "\n", "# shell variables and variable expansion has lots of subtly but lets start simple\n", "i=hello\n", "\n", "# remember shell expansions will happen to the line prior the \"echo\" built in being\n", "# executed\n", "\n", "echo $i # $ -- triggers parameter/variable expansion\n", "\n", "# quoting used to control expansion and splitting\n", "\n", "x=goodbye and farewell\n", "\n", "# single quotes suppresses all expansion and splitting - literal value of each character\n", "# single quotes cannot contain single quotes\n", "x='goodbye and farewell'\n", "echo $x\n", "\n", "x='$i goodbye and farewell'\n", "echo $x\n", "\n", "# double quotes partially suppress expansions but allow some like $\n", "x=\"$i goodbye and farewell\"\n", "\n", "# ; can be used to join commands as a list on the same line\n", "x=\"$i goodbye and farewell\"; echo $x\n", "\n", "# ok we will look at more variable related syntax and expansions later\n", "# lets try out some loops: for, until, while, break\n", "\n", "# for name in word ; do commands; done\n", "for color in red green black brown; do echo \"color=$color\"; done\n", "\n", "# or\n", "for color in read green black brown\n", "do\n", " echo \"color=$color\"\n", "done\n", "\n", "\n", "# the shell also has support for integers and arithmetic\n", "for ((i=0; i<5; i++)); do echo \"i=$i\"; done\n", "\n", "# actually (()) tells the shell to do arithmetic in general\n", "(( j++ )); echo $j\n", "\n", "# shell arithmetic supports comparisons as well\n", "(( j<0 ))\n", "\n", "# special parameter ? is exit code of last command/pipe\n", "echo $?\n", "\n", "(( j>0 )); echo $?\n", "\n", "# shell convention is exit value of 0 is success and non-zero if failure\n", "# be careful might opposite to what you might be used to \n", "\n", "if (( j == 0 )); then echo \"j is zero\"; elif ((j<10)); then echo \"j is under 10\"; else \"j is something else $j\"; fi\n", "\n", "# beyond arithmetic expressions bash support a more general [[]] expression\n", "# return code will still indicate success vs failure but allow non-integer logic\n", "[[ $i == \"hello\" ]]; echo $?\n", "\n", "# special case of logical and and or to conditional combine commands \n", "(( j<0 )) && echo \"j is < 0\"\n", "(( j<0 )) || echo \"j is >= 0\"\n", "# could have written using if\n", "\n", "# conditions are just tests of exit code of command eg. success versus failure\n", "# So any command (internal or external) written to correctly indicate success and failure with exit code\n", "# can be used as condition\n", "\n", "# shell also has case and select statements\n", "```\n", "
" ] }, { "cell_type": "markdown", "id": "b9670b17-98f4-43d3-acab-e393b1be080d", "metadata": { "slideshow": { "slide_type": "skip" }, "tags": [] }, "source": [ "
" ] }, { "cell_type": "markdown", "id": "26afcae6", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Navigating and working with the \"File System\"\n", "**USE TERMINAL and ILLUSTRATE**\n", "- Current Working Directory \n", " - print working directory (`pwd`)\n", "- shell path name expansion : `echo *`\n", "- Externals: \n", " - `ls` daily bread and butter command\n", " - `ls -l` see meta data\n", " - `find` - a even more powerful tool\n", "- Changing Working Directory: `cd`\n", "- Path names: full path vs relative paths\n", " - `.` and `..`\n", " - `~` expansion\n", " - `.` as a way of \"hiding\" files and directories\n", "- Creating directories: `mkdir`\n", "- Removing directories: `rmdir` \n", "- Creating files : `touch`\n", "- Removing files : `rm`" ] }, { "cell_type": "markdown", "id": "ebe73d7b", "metadata": { "slideshow": { "slide_type": "notes" }, "tags": [] }, "source": [ "### File and Directories\n", "
\n", " \n", "``` bash\n", "# Current working directory\n", "pwd\n", "\n", "# What is in the current directory?\n", "# Another expansion - Filename expansion the shell does for us.\n", "echo *\n", "echo /*\n", "\n", "\n", "# ls is a powerfull external command for list contents of directories\n", "ls\n", "man ls\n", "ls -l\n", "ls -lrt\n", "ls -lhrt\n", "ls -1\n", "\n", "# change working directory\n", "#\n", "cd /\n", "pwd\n", "ls\n", "cd home\n", "pwd\n", "ls\n", "cd jovyan\n", "cd /home/jovyan\n", "pwd\n", "cd /usr\n", "pwd\n", "ls -l /var/*\n", "pwd\n", "cd\n", "\n", "# . and ..\n", "ls .\n", "ls ..\n", "\n", "# full path\n", "ls /bin/ls\n", "ls /var/log\n", "\n", "# combining\n", "cd /var/log/../g*\n", "\n", "# short cut to name home\n", "# ~ expansion\n", "echo ~\n", "ls ~\n", "echo ~*\n", "\n", "# . prefix name to hide\n", "ls -a\n", "\n", "\n", "\n", "# creating directoreies\n", "# man mkdir\n", "mkdir ~/bin\n", "mkdir $HOME/tmp\n", "cd ~\n", "mkdir Classes\n", "mkdir Classes/CS\n", "mkdir Classes/CS/400\n", "MYDIR=Classes/CS/400\n", "for ((i=0; i<10; i++)); do echo mkdir $MYDIR/$i; done\n", "\n", "# create an empty file\n", "# man touch\n", "touch foo\n", "ls\n", "rm foo\n", "#for ((i=0; i<10; i++)); do echo touch $MYDIR/$i; done\n", "for d in $(ls $MYDIR); do rm $MYDIR/$d/README.md; done\n", "rm -rf Classes\n", "\n", "# find is also a powerfull command\n", "```\n", "
" ] }, { "cell_type": "markdown", "id": "803bbb51", "metadata": { "hide_input": false, "slideshow": { "slide_type": "skip" } }, "source": [ "## Variables in detail\n", "- has a name and stores a value\n", "- variables are assigned a value with `name=[value]`\n", " - Note cannot have spaces between either name `=` or `=` and value\n", " - by default variable are \"string\" variables\n", " - NULL/empty string is a valid value eg. `x=`\n", " - value of an assignment is subject to the following expansions:\n", " - tilde\n", " - parameter and variable expansion\n", " - quote removal \n", " - if variable has integer attribute then arithmetic expansion is done\n", " - one more attributes that can be assigned with the `declare` builtin\n", " - no need to declare\n", " - eg. assignment to a non-exiting variable creates it\n", " - can mark some variables for export (environment variables)\n", " - \"exported\" variables are copied via UNIX environment to child processes\n", " - Environment a set of strings of the form : \"NAME=STRING\"\n", " - Bash has the kernel copy the current set of exported variables when it launches\n", " a new program. Programs can inspect these values and customize their behavior. (eg. locale)\n", " - `export [=Value]` (can do the same with `declare -x`)\n", "- variable expansion and quotes\n", " - `echo $x $y` vs `echo 'echo $x $y'` vs echo \"$x $y\"`\n", " - `echo \"$x \\$y\"`\n", "- some special unnamed variables (called parameters)\n", " - actually variables are technically parameters with names \n", " - `1` - `n` -- positional parameters expand to ordered arguments of script or function \n", " - `@` and `*` -- expands to positional parameters however with differing expansion rules\n", " - `#` -- expands to number of positional parameters\n", " - `?` -- expands to exit status of last command/pipeline\n", " - others -- `-`, `$`, `!`, `0`\n", "- Advanced: `declare`\n", " - integers \n", " - `declare -i myvar; myvar=myvar+1; echo $myvar` vs not using the declare -i?\n", " - indexed arrays\n", " - associative arrays\n", " - ref variables \n", " - other built-ins that are special forms of `declare`\n", " - `typeset` (same as `declare`)\n", " - `export` (like using `declare -x`)\n", " - `readonly` (like using `declare -r`)\n", " - `+=`\n", " - `unset`\n", "- things that look like variables but are really fancy expansions\n", " - `$` actually triggers three types of expansion \n", " 1. parameter expansion\n", " 2. command substitution\n", " 3. arithmetic expansion\n", " - Parameter expansion \n", " - simple: `$name` or more explicitly `${name}`\n", " - many advanced forms: Here are a few that I constantly use\n", " - setting default value eg. `foo=${foo:-START}`\n", " - delete prefix eg. `foo=ABCA.txt; echo ${foo##*.}`\n", " - delete suffix eg. `foo=ABCA.txt; echo ${foo%%.txt}`\n", " - substitution/deletion eg. `foo=ABCA.txt; echo ${foo//A/O}` or `echo ${foo//A/}`\n", " - Command substitution\n", " - very useful run a command and treat its output as a value\n", " - `foo=$(ls); echo $ls` or `n=$(ls -1 | wc -l); echo $n`\n", " - Arithmetic expansion: `$(())`, or `(())` or let, or if integer types variables \n", " - Shell knows how to do basic integer math and some binary/bit operators \n", " - `++`, `--`, `+`, `-`, `!`, `~`, `**`, `*`, `/`, `%`, `<<`, `>>`\n", " - `<=`, `>=`, `<`, `>`, `==`, `!=`\n", " - `&`, `^`, `|`, `&&`, `||`\n", " - `expr ? expr : expr`\n", " - `=`, `*=`, `/=`, `%=`, `+=`, `-=`, `<<=`, `>>=`, `&=`, `^=`, `|=`\n", " - number formats:\n", " - prefix: `0` - octal, `0x` - hex, `[#base]number` -- eg. `((x=2#101)); echo $x`" ] }, { "cell_type": "markdown", "id": "2409a5df", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "## Shell Commands\n", "See. https://www.gnu.org/software/bash/manual/bash.html#Shell-Commands\n", "2. Breakdown complex commands into simple commands\n", " - complex commands are a composition of simple commands:\n", " - pipelines \n", " - connecting inputs and outputs of commands together using `|`\n", " - `ls -1 /bin | wc -l`\n", " - lists \n", " - one or more pipelines separated by one of: `;`, `&&`, `||` or `&` and terminated by `;` or `;` or `&` or a newline.\n", " - `echo \"about to run ls\"; ls; echo \"done running ls\"`\n", " - `cmd1 && cmd2` : run `cmd2` if `cmd1` returns \"success\"\n", " - `cmd1 || cmd2` : run `cmd2` if `cmd1` returns \"not success\" aka \"failure\"\n", " - `cmd1 & cmd2 ` : run `cmd1` asynchronously in the background and run `cmd2` \"normally\"\n", " - lots more to say about asynchronous commands\n", " - compound commands: Shell programming constructs (these can span lines)\n", " - loops: `until`, `while`, `for`, `do`, `done`, `break`, `continue`\n", " - there are two type of `for` loops\n", " - conditionals: `if`, `then`, `elif`, `else`, `fi`, `case`, `select`, `(())`, `[[]]`\n", " - grouping: `()`, `{}`\n", " - functions\n", " " ] }, { "cell_type": "markdown", "id": "3ca53848", "metadata": { "slideshow": { "slide_type": "skip" } }, "source": [ "## Exercise\n", "\n", "Try and recreate the following\n", "\n", " \n" ] } ], "metadata": { "celltoolbar": "Slideshow", "hide_input": true, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.5" }, "rise": { "autolaunch": false, "enable_chalkboard": true, "footer": "SLS -- Lecture 2", "header": "Unix and Shell Part 2", "height": "100%", "scroll": true, "start_slideshow_at": "selected", "transition": "none", "width": "100%" } }, "nbformat": 4, "nbformat_minor": 5 }