The selection of programs available for the user to run is at the heart of the command line environment. From the outset, the focus of user programs was on file manipulation. The first user programs to be written for &unix; were simple file system commands. After having just implemented the file system, Thompson (1979) wrote a few small programs to manage it — programs to let the user create, copy, edit, list and delete files. These commands alone gave the user a simplistic electronic word processor, but not much more. As more user programs were added to the system, three general categories of program emerged: commands which performed a task with no output (such as copying or deleting a file), those which produced output (such as the directory listing program) and those which transformed output (filters, like the program to sort a file's contents).

The user programs that were developed for &unix; were all small programs that performed a task, and only that task, well. Kernighan and Plauger (1976) expounded on that idea and coined the term software tool. By writing programs to deal with generalities rather than specifics, focussing them on performing a single task well, the user can build up a toolkit from which they can assemble commands to perform complex tasks.

These programs were all written with a certain data format in mind, namely, flat text. It was held that, to maximise portability and usefulness, data should be stored in plain ASCII (the American Standard Code for Information Interchange — a 7-bit code definining a set of alphabetic, numeric, punctuation and control characters), arranged into fields separated by horizontal whitespace and records separated by newline characters .

While certainly an appropriate design decision for the early 1970s, this file format is not always the best today. ASCII is able to represent only a handful of Western, Latin-based languages. There was not much cause for writing with internationalisation and localisation in mind when the standard &unix; user programs were being created. Today, with global networks such as the Internet, languages other than English have to be considered. A character set such as Unicode or the Universal Character Set (UCS) would certainly solve this problem if they were routinely used and had support from the operating system and its user programs.

The rigid record/field format of flat text files also does not accommodate all types of data. A particular example would be information arranged hierarchically. Naturally such information could be stored in flat text, however an inelegant kludge such as recording a node's depth in the tree would have to be stored in its own field. Far better to keep the relationships between the hierarchical data evident due to the file format itself. A data file format such as XML could quite easily be used to store such data, being intrinsically hierarchical. XML also has the advantage of using named fields. A user reading an XML document could easily determine the use of certain fields by their names, whereas in a flat text file, such names are absent, and the user would have to rely on some external documentation. To be maximally useful all text generating programs would have to write their output in XML so they could be used in conjunction with XML filtering programs.

One idea which increases the usefulness of the user programs in &unix; is that of input/output (I/O) redirection. General redirection of process input and output was already available in Multics and it was from here that the idea was borrowed. One of the tenets of &unix; philsophy that Gancarz (1995) mentions is that all programs should be written as filters. Processes under &unix; have the notion of a standard input and standard output. If a program has not been given a filename on the command line, the standard input is where a program typically reads its input from. The standard output, analogously, is where a program would write its output to. Any program that reads data from the standard input, transforms it somehow, then writes that data out to the standard output, is considered to be a filter.

Normally, both the standard input and output of a process is the terminal. Thus, program input comes from the keyboard and program output is written to the screen. In &unix;, both the standard input and output of a process can be redirected to some other file or device. With this, a program can be written to handle just a single input source (the standard input) and be able to handle any file implicitly, by having its input redirected before the program is run; similarly for a program's output. This redirection is performed by the shell, the command interpreter that elicits commands to run from the user. Programs which did not use the standard input and output, and instead relied on hard coded filenames, were definitely of limited use.

While Multics could also handle such redirection, the method used for enacting the redirection in &unix; was far more elegant. According to Ritchie (1979), Multics required the user to use an explicit command to reassign a program's output. For example, to run the list command and redirect its output to a file called listing, the following three commands were issued:

To accomplish the same task in &unix;, running the ls program to generate the listing, just a single command was needed:

Input redirection could be achieved with a similarly simple command:

This command runs the ed editor, taking commands from the file cmds. The syntax of I/O redirection, using '<' and '>', has not changed since its original inclusion in &unix; in 1970.

The mechanism for composing commands is the command pipeline. The concept, which was first seen in the &unix; operating system, is attributed to McIlroy . In 1972 McIlroy proposed the concept of connecting a series of processes together such that the output of one process became the input of the next, the analogy used being that connecting these processes was like screwing together garden hoses. The command pipeline is what enables the reuse of the software tools available to the &unix; user to create complex commands in a single line.

The original syntax used for the pipe concept was similar to the I/O redirection syntax that already existed. A pipeline to sort a file and take the first 10 lines to send to the printer could be written as:

Thus, either a file or a command could be placed after a '>' character to indicate where the output of the previous command inthe pipeline was to go. This syntax, while consistent with the redirection syntax, did allow ambiguous commands to be written. For example, with this syntax there was no way to specify that you wished the output of the sort command to be written to a file called head, as the shell would determine that head was a program and that sort's output should be sent there instead. The syntax was therefore changed to use the vertical bar (or the pipe symbol as it has become known). The above command now could be written as:

It was at this time, after having implemented the pipeline concept, that the developers of &unix; came to the realisation that this simple extension of I/O redirection was the vehicle for program reuse .

When considering the reformulation of the &unix; command line environment to use XML documents, the pipeline concept can remain unchanged. Processes will still write their output to the input of another, only now that output will be in the XML format.

The program that provides an interface to the command line environment is the shell. It is this program that repeatedly prompts the user for a command and then executes that command. The shell is perhaps the one aspect of the &unix; command line environment that has undergone a number of changes since its inception.

The original &unix; shell created at the start of 1970 was very simplistic . All the shell did was read a single command from the terminal, load the specified program over the top of itself, and jump to that new program. The operating system relied on the user program to invoke the exit system call, which would load a fresh copy of the shell into memory and begin its execution. This was not a true multiprogrammed system, though. Only one process could run at one time.

Subsequent developments to the operating system included the fork/exec process control that still exists in todays' &unix;es. The Berkeley time-sharing system developed in 1969 already incorporated this method of process control. With this, the &unix; shell could now handle background processes easily. Background processes are those started with a command such as:

The ampersand indicates that the shell should execute the program in the background and immediately ask the user for another command to run.

From the release of &unix; First Edition in 1971 until the Sixth Edition in 1976, very little changed in the operation of the shell. However, during the development of the Seventh Edition, Bourne (1983) created his own shell, to be known as the Bourne shell. The Bourne shell brought to &unix; shell programming constructs such as iteration and selection that allowed the user to write complex shell scripts. From that point onward, most new shells developed for &unix;, such as the Kornshell and the Z shell , maintained a syntax compatible with the Bourne shell. One popular shell that was significantly different from the Bourne shell was the C shell, csh, written by Bill Joy for inclusion into the Berkeley Systems Distribution of &unix; . The C shell, as its name implies, lets the user run commands with a syntax reminiscent of the C programming language .

Attempts to vastly redesign the &unix; shell have been made. One notable design is that of a functional shell, where commands are entered in a form similar to those used in a functional programming language. These shells, such as McDonald's (1987) fsh and the Scheme shell , scsh, allow the user to compose commands as if they were functions.

All of these shells, however, have still been designed with the assumption that the majority of user programs that are to be composed use flat text for their input and output. The shells can let program output be written verbatim to the terminal, since flat text is eminently human readable. If, however, user programs are to be designed to read and write XML documents, program output is not immediately appropriate for the user to read. XML documents are readable by humans — they do not contain any illegible control codes as some binary formats do — but the information in these documents should be converted into a form that is easy for the user to read. Since the user will be interacting with the shell using a text based terminal, program output destined for the terminal should be transformed into plain text by the shell. Making this final transformation for the user the responsibility of the shell means that user programs do not have to worry about checking whether their output is to be piped into another program, redirected into a file or written to the terminal.