What we can learn from Unix
The Unix Philosophy is a wonderfully cohesive way to thing about program execution and composition. Its beauty relies in its simplicity, somewhat unfortunately this is also tied to the fact that Unix is dealing with software on the operating system level. We're going to explore what the Unix Philosophy is and what it allows us to do when working in a Unix shell. All the while exploring how Unix can guide our attempts to create simpler programs at a higher level of the stack.
What is the Unix Philosophy?
The Unix Philosophy as described by Doug McIlroy - Inventor of the Unix pipe - is the following:
- Write programs that do one thing and do it well.
- Write programs that work together.
- Write programs that handle text streams, because its a universal interface.
The core focus of unix is really simplicity. In part this is possible because unix is focused on the operating system level and is therefore dealing with more simple representations, mainly text. As an operating system unix can make more assumptions in regards to what protocols it can use and therefore exert more control over the content.
What is perhaps more critical is that unix commands focused on manipulating text
streams. All traditional unix commands work by taking some text as input from a
stream, and responding back with more text. For example if I run
ls -l in my
blog directory, I'll get back the following:
404.html CNAME LICENSE README.md _config.yml _data _drafts _ideas _includes _layouts _pages _posts _projects _refs _sass _site atom.xml css favicon.ico public
This is just text. Which means I can count the number of lines in it and know how many folders or files I have here, or I can replace some of the text etc. This means that any command can take the input of any another command or be the output to any other command. This is particularly powerful as any possible composition can be made with these small building blocks. This is the most essential ingredient to making unix as useful as it is.
Benefits when working in the shell
wc do very simple tasks. For those who aren't
ls displays the files in your current directory,
rm removes a file
at the specified path and
wc counts characters, words and lines in a file. For
example if you want to know how many files or folders you have in a directory
you can run the following at the command prompt:
$ ls | wc -l
ls gives you a list of files line by line, and
wc -l tells you the
number of lines in the output of
ls. The key idea here is that you can pipe
the output of one program into the output of another. This is what it means to
pipe one thing into another.
If I want to do something more elaborate that can be done pretty easily as well. Say I'm doing some writing and I want to know how many unique words I've used in a post.
$ cat unix.md | tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq | wc -l
Another even more complicated example. Say you want to know which words in your post you've used a lot of.
$ cat unix.md | tr -cs A-Za-z '\n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed 10q
This will print out a listing of the 10 most common words and how often they've
been used in the file
unix.md. There is a bit of arcane knowledge here, but
that isn't necessarily bad. The
tr command just replaces whats on the left
with whats on the write. So
tr -cs A-Za-z '\n' is taking anything that is not
a letter and replacing it with a new line.
For comparison here is the equivalent Python script:
from collections import Counter text = "" with open("unix.md") as f: text = f.read() c = Counter(map((lambda s : s.lower()), filter((lambda s : s.isalpha()), text.split(" ")))) for word, num in c.most_common(10): print(num, word)
It's worth noting that the python script will not count words that have a comma
after them or some other punctuation. It would not count the word
it isn't strictly alphabetical. In order to could it you would have to bring
regex or the python string translate method into the mix, but it seemed like
Can we apply these concepts higher up the stack?
If we know that unix principles are useful, and make building systems and programs easier, how can we apply those insights higher up the stack?
If we write smaller functions that focus on a single task the quality of those functions will go up, the overall quality of the library will increase, but the amount of work an individual will have to put in to learn that library will also increase. Furthermore, the library will probably not be as successful as libraries that give all in one magic methods for common tasks.
Unix requires very active users, who are willing to learn and comb through man pages. Developers are of course willing to do this to an extent, but what they really want are magic boxes to solve their problems. Which if we're honest, who can blame them? The issue is that you end up relying on code that you don't understand and therefore can't debug. This obviously isn't a good situation to be in, and is why full stack developers are sought after. Unix forces you to learn, which slows things down, harms adoption rates, but also produces better developers.
Well we know that writing commands or functions that can be composed is useful, and in order to compose them the functions themselves must generally be stateless. That is if you run a command twice with the same parameters you ought to get the same results. This is true for unix commands and that lesson is being applied daily by functional programmers.
One of the areas in which I think we need to spend more time on is a expansion of text streams as the universal interface. I do think that text is incredibly useful, and very close to being a universal interface outside the unix ecosystem. Where text streams fall short, at least where unix is concerned is when you're working with data formats that are highly nested. For example JSON or XML.
As a result most developers spend the vast majority of their time dealing with
parsing data streams and working with text. Of course, the lack of support in
unix is not the only issue, but if we look at the manner by which unix modeled
text streams they contain characters, which compose the entirely of the stream,
as well as special delimiter characters, such as
\t. As a result
common operations focus on working with characters, lines or words, as defined
by the special delimiters or whitespace. There are good simple command line
tools like jq for working with JSON in the command line which aim to fix
this issue. Essentially the issue boils down to mapping between two different
representations, which is not a simple task for a machine to do automatically.
There's a lot to be learned for unix, primarily to keep it simple, and keep it focused. The result is software that is easy to understand, and relatively simple to use.