The selection of programs available for the user to run is at the heart of the command line environment. From the outset, the focus of user programs was on file manipulation. The first user programs to be written for &unix; were simple file system commands. After having just implemented the file system, Thompson (1979) wrote a few small programs to manage it — programs to let the user create, copy, edit, list and delete files. These commands alone gave the user a simplistic electronic word processor, but not much more. As more user programs were added to the system, three general categories of program emerged: commands which performed a task with no output (such as copying or deleting a file), those which produced output (such as the directory listing program) and those which transformed output (filters, like the program to sort a file's contents).

The user programs that were developed for &unix; were all small programs that performed a single task, and only that task, well. Kernighan and Plauger (1976) expounded on that idea and coined the term software tool. By writing programs to deal with generalities rather than specifics, focussing them on performing a single task well, the user can build up a toolkit from which they can assemble commands to perform complex tasks.

These programs were all written with a certain data format in mind, namely, flat text. It was held that, to maximise portability and usefulness, data should be stored in plain ASCII (the American Standard Code for Information Interchange — a 7-bit code definining a set of alphabetic, numeric, punctuation and control characters), arranged into fields separated by horizontal whitespace and records separated by newline characters .

One idea which increases the usefulness of the user programs in &unix; is that of input/output (I/O) redirection. General redirection of process input and output was already available in Multics and it was from here that the idea was borrowed. One of the tenets of &unix; philsophy that Gancarz (1995) mentions is that all programs should be written as filters. Processes under &unix; have the notion of a standard input and standard output. If a program has not been given a filename on the command line, the standard input is where a program typically reads its input from. The standard output, analogously, is where a program would write its output to. Any program that reads data from the standard input, transforms it somehow, then writes that data out to the standard output, is considered to be a filter.

Normally, both the standard input and output of a process is the terminal. Thus, program input comes from the keyboard and program output is written to the screen. In &unix;, both the standard input and output of a process can be redirected to some other file or device. With this, a program can be written to handle just a single input source (the standard input) and be able to handle any file implicitly, by having its input redirected before the program is run; similarly for a program's output. This redirection is performed by the shell, the command interpreter that elicits commands to run from the user. Programs which did not use the standard input and output, and instead relied on hard coded filenames, were definitely of limited use.

While Multics could also handle such redirection, the method used for enacting the redirection in &unix; was far more elegant. According to Ritchie (1979), Multics required the user to use an explicit command to reassign a program's output. For example, to run the list command and redirect its output to a file called listing, the following three commands were issued:

To accomplish the same task in &unix;, running the ls program to generate the listing, just a single command was needed:

Input redirection could be achieved with a similarly simple command:

This command runs the ed editor, taking commands from the file cmds. The syntax of I/O redirection, using '<' and '>', has not changed since its original inclusion in &unix; in 1970.

The mechanism for composing commands is the command pipeline. The concept, which was first seen in the &unix; operating system, is attributed to McIlroy . In 1972 McIlroy proposed the concept of connecting a series of processes together such that the output of one process became the input of the next, the analogy used being that connecting these processes was like screwing together garden hoses. The command pipeline is what enables the reuse of the software tools available to the &unix; user to create complex commands in a single line.

The original syntax used for the pipe concept was similar to the I/O redirection syntax that already existed. A pipeline to sort a file and take the first 10 lines to send to the printer could be written as:

Thus, either a file or a command could be placed after a '>' character to indicate where the output of the previous command inthe pipeline was to go. This syntax, while consistent with the redirection syntax, did allow ambiguous commands to be written. For example, with this syntax there was no way to specify that you wished the output of the sort command to be written to a file called head, as the shell would determine that head was a program and that sort's output should be sent there instead. The syntax was therefore changed to use the vertical bar (or the pipe symbol as it has become known). The above command now could be written as:

It was at this time, after having implemented the pipeline concept, that the developers of &unix; came to the realisation that this simple extension of I/O redirection was the vehicle for program reuse .

The program that provides an interface to the command line environment is the shell. It is this program that repeatedly prompts the user for a command and then executes that command. The shell is perhaps the one aspect of the &unix; command line environment that has undergone a number of changes since its inception.

The original &unix; shell created at the start of 1970 was very simplistic . All the shell did was read a single command from the terminal, load the specified program over the top of itself, and jump to that new program. The operating system relied on the user program to invoke the exit system call, which would load a fresh copy of the shell into memory and begin its execution. This was not a true multiprogrammed system, though. Only one process could run at one time.

Subsequent developments to the operating system included the fork/exec process control that still exists in todays' &unix;es. The Berkeley time-sharing system developed in 1969 already incorporated this method of process control. With this, the &unix; shell could now handle background processes easily. Background processes are those started with a command such as:

The ampersand indicates that the shell should execute the program in the background and immediately ask the user for another command to run.

From the release of &unix; First Edition in 1971 until the Sixth Edition in 1976, very little changed in the operation of the shell. However, during the development of the Seventh Edition, Bourne (1983) created his own shell, to be known as the Bourne shell. The Bourne shell brought to &unix; shell programming constructs such as iteration and selection that allowed the user to write complex shell scripts. From that point onward, most new shells developed for &unix;, such as the Kornshell and the Z shell , maintained a syntax compatible with the Bourne shell. One popular shell that was significantly different from the Bourne shell was the C shell, csh, written by Bill Joy for inclusion into the Berkeley Systems Distribution of &unix; . The C shell, as its name implies, lets the user run commands with a syntax reminiscent of the C programming language .

Attempts to vastly redesign the &unix; shell have been made. One notable design is that of a functional shell, where commands are entered in a form similar to those used in a functional programming language. These shells, such as McDonald's (1987) fsh and the Scheme shell , scsh, allow the user to compose commands as if they were functions.

The traditional ls program takes a list of files on the command line and displays information about those files. If a file on the command line is a directory, information about the files inside that directory are given instead. If no files are given on the command line, just information about the files in the current directory is given.

Without any formatting options, ls outputs just a list of the filenames. If the output is destined for a terminal, these files would typically be lined up in coumns to save space on the screen. If the output is going to a file or another process' standard input, the filenames would be listed one per line. Such output is good for parsing, as there is no real formatting going on here.

One of the most commonly used options, however, does force some formatting to be done. The -l option causes ls to output a long listing with details about each file, such as its permissions, size and last modified date. This information is arranged into fixed width columns, regardless of whether the output is destined for the terminal or another process' standard input. An example of this output is shown below:

The first thing output is the number of disk blocks taken up by the files which are listed. Then comes a line for each file, detailing the permissions in symbolic form, number of hard links, owner, group, size, last modified time and name of the file. One particularly nasty problem with this output is that the custom for using whitespace to separate fields has been broken. The last modified time actually has spaces embedded in the field. "Oct 22 23:40" should be just the one field of this record. This means that it is somewhat difficult to extract this information from the record. One way to do it would be to use the cut program, which can select the text from each record between two given columns. Obviously this is going to need the user to count the characters in this output to determine where the last modified time begins and where it ends. Really the user would prefer to issue a command to just "extract the last modified time" rather than "extract characters in columns 40 through 52".

It can also be seen that the ls program changes the format of the last modified date depending on the actual date. If the last modified date is within about ±6 months of the current date, the month, day of month and time will be displayed. But if the last modified date falls outside this range, the month, day of month and year will be displayed instead. This is perhaps not such a bad idea when considering what the user wants to see. If a document has been changed over a year ago, it is more important to mention the year of modification than the time of day. For extract information, however, this is not so good. Ideally, the full time would be listed for every file, so that no information is lost.

The ls program also has another mode, enabled by using the -R switch. This causes ls to recurse through directories to list information about files in a whole directory tree. An example of the traditional ls -R output is listed below:

The output for the recursive directory listing is split into sections. Each section is for one directory, and it starts with a header comprising the name of the directory followed by a colon. Each file is the directory is then listed after this header. A blank line separates each directory. This output format breaks the customary flat text format even more than the previous example did. Now instead of there being a file per line, any program parsing this output must be careful of the headers and the blank lines. For a human, this output format is fine, as it is a reasonable way to flatten the hierarchy of the directory tree for display on the terminal. The inherent structure in the directory tree has been lost, though, and it is non-trivial to extract information about a file's position in the directory tree.

To overcome these shortcomings in the traditional ls program, the new ls will produce an XML document with an element for each file being listed. Each file element in the output will have a number of attributes, used to hold the details of that file. Also, each file element can contain other file elements, thereby preserving the structure of the directory tree.

Here is the above ls example with the new output format (some attributes have been elided for brevity):

The output document uses the namespace urn:ns:clm.xml-unix.ls. The specific namespace that was chosen here does not matter, though it is important that it is unique. The namespace will be used by the shell after the ls program is run to determine the correct transformation to perform on this output to make it human readable. The rec attribute is used in the root element to inform the transformer that a recursive directory listing format is required when it comes time display it on the terminal.

The last modified time is now bundled up in a single attribute, to facilitate its extraction. In fact, for the last modified time, two attributes are used to represent it. The first, mtime, holds the &unix; time — this is the number of seconds since the &unix; epoch, 1 January 1970. The second attribute is human-mtime, and this holds the human readable form of the last modified time. It is this field which will be output to the terminal in the directory listing, as it is much easier for a human to understand than the &unix; time.

The ps program reports the status of processes running on the system. ps, while differing in behaviour across different &unix; systems, typically has two modes of process selection. By default, the ps command will only report processes which are running in the same session as itself. Usually this results in the processes which have been started under the current login shell being listed. The -e option instructs ps to list every process running. (There exist other criteria by which to select processes to list, such as by process ID or by user, however these are not as interesting as the major two mentioned, and can be reproduced using a filter on the data.)

Which properties of processes to report is the other main aspect of ps. By default, just four fields are shown — process ID, controlling terminal, running time and command. Using the -f flag asks for a full listing. A full listing comprises the username, process ID, parent process ID, percentage CPU usage, start time, controlling terminal, running time and command fields. If neither of these two alternatives is appopriate, the -o argument may be used to specify which fields should be displayed.

Two of these fields contain embedded spaces which make them harder to extract from the flat text output — the lstart field, which contains a long format date, and the command field, which holds the full command line for the process. Also, the command field and the wchan field, which contains the system call that process is currently running in, will be truncated if they are not the last column in the output.

A simple XML file is used to store all of the relevant information about a process in the new model ps program. An example output document from running ps -f is given below (very many attributes omitted):

To signify that the "full" listing format is required, a full attribute is added to the root element of the document. Also, an sid attribute is added. This contains the process ID of the session leader of the ps process. This is helpful for the transformation process the shell will perform later, in determining which processes should be displayed.

Thus, although only some processes will be written to the terminal, all process details are included in the output. Selecting which processes to display is a formatting issue, and should be delayed until as late as possible.

The df program is a relatively simple tool to report the amount of free disk space on each of the currently mounted filesystems. Some systems, such as Linux, have a number of virtual filesystems which don't actually take up any space. These filesystems are by default not listed by df, unless the -a option is given.

df does not have much wrong with its output. It is straightforward and easy to parse. The only thing the user must be wary of is the header on the first line.

An XML version of the output of running df -a is shown below:

Like both ls and ps, a formatting option (in this case, -a) is handled by storing it as an attribute in the root element of the document.

The cat program is a very simple one in the traditional &unix; command line environment. Its job is simply to concatenate files and write them out to standard output. The concatenation is trivial; since flat text files use line based records, the program needs only to write out each file in succession to standard output for them to be joined together.

Unfortunately, such simple behaviour will not work with XML documents. This is because outputting one document after another will not result in a well formed document. Well formed documents must contain exactly one root element. Thus, cat must do something else.

If it is assumed that records are stored as elements directly under the root element of the document, as outlined in the model description in section 3, then there is a fairly simple means of concatenating two XML documents together. All that must be done is to take the second level elements from the source documents and insert them just before the closing tag of the destination document's root element. For example, if the file doc1.xml contains:

and doc2.xml contains:

then running the command cat doc1.xml doc2.xml should result in the following document:

Note that the implementation here does not provide any checking to ensure both documents are in the same namespace or have the same root element. Traditional &unix; cat doesn't check for similar file formats either; it just outputs the files with no regard to if they "should" be allowed to be joined together.

If the important data is elsewhere in the source document, however, there needs to be some way of specifying which elements are to be selected. Similarly, if the destination document shouldn't have its root element appended to, there must be a way of specifying the insertion point for the new data. The cat program presented here has a command line argument --from which allows the user to give an XPath location specifying which elements from the source document are to be copied. Analogously, there exists a --to argument which should be followed by an XPath location indicating where in the destination document the source documents' elements are to be inserted.

The default --from XPath location is /*/node(), which refers to all of the nodes (that is, elements, text nodes and attributes), underneath the document's root element. The default --to XPath location is /*, which refers to the root element of the destination document.

To demonstrate these options, consider the file doc3.xml to contain:

and doc4.xml to contain:

If the command cat --to /html/body doc3.xml --from '/html/body/*' doc4.xml is then issued, the resulting document should be:

As encoding is an important issue with XML documents, unlike with traditionally 7-bit ASCII flat text files, the cat program also has a command line argument to specify which encoding should be used for output. Thus, to transcode an XML document to EBCDIC-US, the command cat --encoding EBCDIC-US is used. libxml2 can be compiled with support for the libiconv internationalisation library. Running the command iconv -l at the command line produces a list of encodings which cat should therefore be able to handle.

The grep program is one of the most useful of the traditional &unix; command line environment filters. Its purpose is to select records (i.e., lines, in the flat text model) which match a given regular expression. As with cat, grep must be told where in the XML document it should be operating.

The grep program presented here takes two parameters to control where the search takes place. Firstly, it has a --context command line argument. This tells grep what each record of the XML document is. By default, this will be the XPath location /*/*, the elements directly underneath the document's root element. Secondly, grep can be passed a --node argument. This controls, relative to each node that matches the context XPath location, which nodes should be compared against the regular expression. The default node XPath location is //text(), that is any text node underneath the context node.

For the regular document structure mentioned earlier, the default XPath locations work well. For example, assuming that the file doc5.xml contains:

then issuing the command grep 'a[^ ]' doc5.xml will produce the following XML document:

To demonstrate the use of the --context and --node arguments, assume the file doc6.xml contains the following:

Then, the command grep --context /data/entries/entry --node @name 'entry[12]' doc6.xml can be used to select the first two entries:

Notice that the two entry elements are returned still contained in the entries element. This is because the elements which match the regular expression are checked to see if they have a common parent, and since they do, they are placed within it in the output document.

The traditional sort program rearranges the records in a file, sorting them by a particular field. The sort key is specified by giving a field number or range of characters. The -r command line argument is used to reverse the sort order, and -n is used to signify a numeric (rather than lexicographic) sort.

Just as with cat and grep, new version of sort defaults to sorting the elements directly beneath the document's root element. If this is not appropriate, a --context and --node command line argument can be given to select which elements are to be sorted, and on what they are to be sorted, respectively. By default, sorting is done just on the text nodes beneath each context node.

Below is a demonstration of how the new model sort works. If the file doc7.xml contains:

and the command sort doc7.xml is issued, the resulting XML document will be:

A different sort key can be specified. Running the command sort --context '/foods/*' --node @id doc7.xml results in:

The sort command can also take an XPath expression (not just an XPath location) to sort on. This makes it possible to, for example, sort based on the name of the element being sorted. Executing sort --context '/foods/*' --expr 'name()' doc7.xml produces:

As can be seen from this example, the sort is stable. The two fruit elements are considered equal in the sorting, as are the two staple elements. Their relative ordering is the same as their relative ordering in the original document.

The goal of this task is to determine which filesystem has the most free disk space. In order to determine which filesystem has the most free space, the df command must of course be used. Both the traditional and the new df print the same output to the terminal, but of course underneath it is all different.

To select the filesystem with the largest amount of free space, the sort command can be used to arrange the records such that the last record is the one with the most free space. In the traditional system, the user must be mindful of the header df also outputs. Since it is part of the flat text output, it must be avoided. The tail +2 command can be used to remove that first line. Subsequently, the records can be sorted. Since the traditional sort command uses field numbers to select the key, the sort -n +3 command is used to sort the records numerically by the fourth column. Finally, the tail -1 command can be used to select the last record from the sorted list. That record will be the one with the highest amount of free space.

In the new model, the header does not form part of df's output, so it does not have to be avoided as must be done in the traditional system. The output of df can be piped straight into sort. Inspecting the output format of df, it is apparent that the free attribute contains the number of free blocks on a filesystem. Thus, the command df's output must be piped through is sort --numeric --node @free. Once the records are sorted, the last one can be chosen as with traditional system, using tail -1.

In summary, the traditional &unix; command to find the filesystem with the most free space is:

The command needed in the new model is

The new model command, while taking more keystrokes to type, uses one fewer pipeline stage to generate the output. Also, it is more intuitive than the traditional &unix; command. The traditional &unix; command requires a numeric reference to the sort key, while the new model command uses a symbolic key name. The sort command should thus be easier for the user to write.

This goal of this task is to display only the process ID and the command line of the process which has used the most CPU time.

Obviously, the ps command will need to be used in this command pipeline. To begin with in the traditional system, all processes are selected for display by the ps command. Also, the relevant fields (process ID, CPU time and command line) are also selected. As with the previous task, the header must be stripped off. Luckily, the GNU ps program being used has a --no-headers command line option. Thus the ps command used is ps -e -o pid,time,command --no-headers.

Once all of the records are available, they must be sorted based on their running time. This is done with the sort command, specifying the second field as the sort key. The command used is sort +1. The last record in the output will now be for the process with the longest CPU time. It can be selected with the tail -1 command.

Finally, since the goal of this task is to display only the process ID and the command line, the CPU time must be removed. Unfortunately, since the command line could have spaces in its field, a field based solution (such as using awk) is not possible. Instead, sed can be used to match the time a remove it from the line. The command to do this is sed 's/ ..:..:..//'.

Building the pipeline in the new system starts similarly. However, when selecting which fields to display, we can omit the CPU time. This wasn't possible in the traditional model, since if the CPU time wasn't included in the initial ps output, its value could not be used as the sort key. With the new model, however, all of the information about each process is included in ps's output document. It is just that the process ID and command line can be registered as the only fields to be displayed when the shell finally transforms the document for output to the terminal. Just like in the previous task, the header is part of the formatting and thus is not present in the XML output document of ps. Thus it doesn't not need to be explicitly excluded. The initial command in the pipeline is therefore ps -e -o pid,command.

Looking at the raw XML output from the ps command, it can be seen that the time attribute holds the CPU time field to be sorted on. So, the next command in the pipeline is sort --node @time. The last record in the sorted list, which has the greatest CPU time, can be selected with tail -1. Since the formatting options given to ps initially preclude the CPU time from being displayed, there is no need to remove it as in the traditional model.

Comparing the two models, the traditional model solution resulted in the command:

while the new model solution produced the command:

The new model gives the user a much cleaner and shorter command to produce the correct output. Again, the command is more intuitive (especially the sort command) than the equivalent traditional model command. The traditional model command needs more text processing to massage the data into an acceptable form.