The Rise of The Graphical User Interface

INTRODUCTION

One of the most important developments in information technology over the past ten years or so - quite apart from the massive improvements in hardware technology - has been the graphical user interface (GUI). For most people it has been a positive innovation, but for some - particularly those who are blind or visually impaired - it has been a rising threat as a barrier to the technology. Now that such interfaces have matured and become the norm, adaptations have been developed and perhaps that threat is not as bad as it was once feared to be. This paper describes the development of the GUI, why it is so significant and discusses whether it has been "tamed" with respect to use by people with visual disabilities.

The advent of the microcomputer had a profound effect on information technology. Now that computers were being mass produced, the more units that could be sold the cheaper they would be - and the cheaper they were the more people could afford to buy them. Cost, however, is not the only barrier to the purchase of computers -- they also had to be usable. So it was effectively with the advent of the microcomputer in the early 1980's that the discipline of human-computer interaction (HCI) became established.

Undoubtedly the greatest innovation in the field of HCI has been the adoption of the graphical user interface (GUI). For the vast majority of new users the graphical user interface is much more usable than anything that has gone before - as long as they can see the screen. The fundamental problem is that sight is very efficient. This is often summarized (somewhat loosely) by saying that sight has a very broad bandwidth; what sight can do is take in a lot of information and furthermore it has control over that information flow, allowing the person to focus on what is important and filter out that which is not. Much of the power of GUIs comes from the fact that they make the most of this ability; they present a lot of information on the screen but can rely on the power of sight to cope with all of it. None of the non-visual senses has that same power, that bandwidth.

ORIGINS OF THE GUI

The reasons why GUIs are easier to use are many and have been explored and documented in many research papers and books. (Chapter 4 of Dix, Finlay et al., 1993 includes a discussion). To most sighted users the question never arises. Most people who have tried a GUI have found it much easier to use than the alternatives. (The general exception to this is expert users of traditional interfaces who often find GUIs limiting and long-winded). The components of the interface similarly need no description to sighted people, they can simply be shown them. For people who can never have the experience of seeing or using a GUI the concepts are difficult to describe, but anyone who needs such a description might consult Morley (1995).

The history of graphical user interfaces (GUIs) goes back to the 1970s. Project Smalltalk was established at Xerox Palo Alto Research Center (Parc) which attempted to look into the future. The idea was to assume that in the future computing power would be abundant and inexpensive. How could the best use be made of the power available? Two influential developments resulted: object-oriented programming and the graphical user interface. Much of the inspiration behind the GUI came from Alan Kay's vision of the "Dynabook" - a powerful, easy-to-use computer which would be a portable and ubiquitous source of information, a vision which is almost frighteningly familiar in this age of laptops and personal organizers.

The Star GUI was implemented on the Alto system and included most of the elements that are so familiar today: windows, icons, menus, etc. (Smith, Irby et al, 1982). This was still a development project and it was Apple who went on to commercialize the ideas. The story is that Steve Jobs, one of the co-founders of Apple, was invited down the road from Cupertino to Palo Alto to see what was going on at Parc. There he saw Star and was immediately struck by its possibilities. He went back to Apple determined that the next generation of machines they would market would have the same style of user interface as the one he had seen at Xerox.

Jobs proceeded to hire many of the developers from Xerox, including Alan Kay, who used their expertise to design the new Apple system. The first incarnation was the Lisa (named after Jobs' secretary). This was for its time a very powerful microcomputer and had a graphical user interface. The Lisa was not a commercial success, but Jobs was unperturbed and went on to develop the Macintosh (Macintosh being his favourite variety of apple).

The Macintosh was released in 1984. It consisted of a keyboard and mouse attached to a box which contained a 9-inch monochrome screen and a single floppy disc drive. The basic version had 128 kbyte of RAM, though the "Big" Mac was available with 512 kbyte. For its time it was under-powered and over-priced, compared to available PC clones. What it had that was different was its user interface. At that time the style of interface was usually referred to as "Wimp," derived from the components "window, icon, menus and pointer," but that term seemed to die out due to its negative connotations and was replaced by GUI - or even "gooie."

FIRST ATTEMPTS TO TACKLE THE GUI PROBLEM

In 1984 I was researching for my Ph.D. at the Open University in England. Tom Vincent was a member of the university who had already done a lot of work on making computers accessible to blind people, mainly through the use of speech synthesizers. He had already recognized the potential problem posed by the increasingly visual nature of user interfaces and, having talked to him, I made a major shift in my research direction (from computer-aided learning) and started out to tackle the GUI. At that time many people saw the Macintosh as a passing fad. They believed it was all very well as a means of making word processing easier for secretaries, but "real" computer users would never want to mess around with mice and menus and all those gimmicks. In the same manner, many blind computer users did not see any threat. They thought that command-line interfaces, such as MS-DOS and Unix would always predominate. Some of us thought they were wrong - but it was hard to find corroboration. Eventually I was able to find a quote to include in my thesis from a not-well-known futurist, Zachmann (1987): "The future lies with a graphical windowing interface, mouse cursor control, pull-down menus, dialog boxes, and the like" and that computers based on such interfaces "are destined to take over the IBM PC and compatible world as well." (pp. 13-14).

Scadden (1984) was one of the first to recognize the potential threat: "If such visually-oriented computer equipment were to become dominant in the microcomputer marketplace, blind and visually impaired people would lose much of the newly acquired equality unless... alternative approaches were available (p. 399)."

My approach was to make the adaptation entirely in software. I concentrated on the use of sounds (speech and non-speech) and on making the mouse usable in a non-visual interaction. I realized that the ultimate requirement would be for screen-reader-like adaptations, which would adapt a variety of applications, but for the present I tackled the simpler problem of writing one particular application, a word processor. The result was Soundtrack. To the best of my knowledge Soundtrack was the first program which embodied the essential characteristics of a graphical user interface which was usable by blind people and it is still one of the few which can be used through a mouse.

Soundtrack had an unconventional screen layout which consisted of eight "auditory windows." Each window had a sound associated with it which was heard as the cursor was moved into that window. The sounds were arranged according to their musical pitch. The windows had names too, and the name could be elicited (in synthetic speech) by pressing the mouse button within the window. Each window had a particular role; four of them corresponded to menus, for instance. A second level of interaction was reached by double-clicking the mouse. That would cause the current window to become subdivided into a number of components (auditory items). Each of those had a sound and a name, just as the windows had, as well as an action, which was performed if the user double-clicked the item.

Further details of Soundtrack can be found in Edwards (1987) and Edwards (1989). It was never intended to compete with other word processors, it was only a research vehicle. Copies were made available and feedback was invited, but I do not believe anybody took to using it regularly. Nevertheless, it achieved its objective of demonstrating one way of making a GUI accessible through the fairly extensive evaluation I carried out. As a piece of academic research I was bound to be candid about its weaknesses and I think it is a bit unfortunate that it was the weaknesses that people picked up on. Soundtrack did demonstrate that mouse-based interactions could be transformed into a non-visual form. The main problem with using it was that it imposed an extra load on the user's memory. For instance, whereas most of my evaluators could easily remember that the delete command in their word processor was Ctrl-X (or whatever), they now had to remember that the Cut command was in the Edit menu and the Edit menu was the second one on the top row, and so on.

I and colleagues at the University of York took this work a little further, continuing to investigate ways of using sounds to guide a blind user through a two-dimensional space with a mouse. That work is written up in Pitt and Edwards (1991) and Pitt and Edwards (1995). Apart from that, our research has moved off in other directions and we have left others to continue with the GUI problem - as described below.

DEVELOPMENT OF ADAPTATIONS

The first commercial Macintosh adaptation was released in 1988. That was Outspoken, from Berkeley Systems Inc. It maintained the screen reader style of adaptation in that it worked with a variety of applications (though sadly not all - mainly for technical reasons). The designers took a different approach from that of Soundtrack in that they got around the need to use the mouse. The basic idea is that whatever the cursor is pointing to on the screen is spoken out. One can move the cursor using the mouse in the usual way, but when working non-visually and relying on the speech output that is not really feasible. Therefore alternative control of the cursor is available through the keypad. For instance, four keys are used to move the cursor up, down, left and right (8, 2, 4 and 6 respectively). Other keys have other roles, so that the 5 key corresponds to the mouse button, another takes the cursor directly to the menu bar and so on.

The developers of Outspoken performed a remarkable job since they delved around in the operating system and succeeded in extracting all sorts of information not intended to be accessible to other software. Unfortunately not all the information required to make a good adaptation was accessible. A simple example is "greyed-out" ("disabled" in the language used in Morley, 1995 ) menu entries. Commands which are currently inappropriate cannot be selected and this is signified by their being displayed in a grey colour. For instance, in a word processor with no document open, the CLOSE command cannot do anything and would be greyed out. However, the fact that a menu item is grey is not accessible to Outspoken and so is read out in the same manner as a black, active item. Thus such an item can apparently be selected and there is often no indication to the user that the command has had no effect. Further critical evaluation of Outspoken can be found in Edwards (1991) and Edwards (1995). So, Outspoken was important historically and technically as the first screen reader for a GUI but its effect in terms of increasing accessibility has been limited.

The Macintosh was the first widely available GUI and Outspoken the first non-visual adaptation. Meanwhile in the PC world there were various attempts to promote graphical interfaces. It was not until Microsoft, the originators of DOS, released their Windows interface that things really started to change. Slowly the market has drifted from people running the majority of their applications under DOS to the situation where Windows has taken over as the default. As that has happened the community of blind computer users has begun to take notice - and to worry.

There have been a number of responses. Commercial developers have produced adaptations. The first to appear on the market was Window Bridge from Syntha-Voice in Canada. There has also been a considerable amount of research into solving the problem.

In 1991, the European Commission established the TIDE initiative (Technology for the Integration of Disabled and Elderly people). One of the first projects it funded was GUIB (Graphical and textual User Interfaces for Blind people). As with all such projects, this was an international collaboration involving partners in several European countries.

The initial approach of GUIB was to investigate the use of a variety of input and output media. For output there was an extensive Braille display (some 200 Braille cells in all), three-dimensional sounds, and speech. For input there was the keyboard, a touch-sensitive pad, routing keys associated with the Braille cells and other cursor keys on the Braille display. (Further details can be found in Crispen and Petrie, 1993). GUIB progressed into the second round (Bridging Phase) of the TIDE initiative and eventually resulted in the Windots screen reader which is now commercially available from Papenmeier. Windots relies on the use of a Braille display (though a more conventional 80-cell one) but a speech-only version is also due for release.

The original philosophy behind GUIB acknowledged the bandwidth problem, mentioned earlier. If no one of the non-visual communication channels has the same capacity as the visual one being replaced, then perhaps several in combination will substitute more effectively. However, in the end, other considerations such as cost have led to the release of versions of the GUIB screen reader which use a single channel (speech or Braille). I fear their usability will not match that of the development prototype.

IBM's newest personal operating system is OS/2, written for its PS/2 range and has a graphical user interface. A screen reader was developed to work with it. IBM came up with the name Screen Reader to describe this product - and trademarked it. Screen Reader is something which grew out of a student project and for long enough was not intended to be a commercial product. The advantage that Screen Reader's developers had over third party adapters of other systems was that they had access to the code for the operating system - and the people developing that code. This meant that they were able to find out exactly how to access the system information and even request hooks to be inserted in future releases to facilitate their access. Descriptions of Screen Reader and its development can be found in Schwerdtfeger (1991) and Thatcher (1994). OS/2 never seems to have challenged Microsoft's operating systems in the marketplace and the number of Screen Reader users is small - though as Windows can be run within OS/2 this can be another means of accessing Windows.

Another GUI which has grown in popularity, X-Windows, runs on Unix systems. This originated from the Massachusetts Institute of Technology (MIT) and as such developed in a very open manner. Administration of the development of X has been taken over by the X-Consortium which has been an important factor in getting X made accessible. Beth Mynatt and her colleagues at Georgia Tech have tackled the accessibility of X through their Mercator screen reader. It has been most important that they have been able to collaborate with the X-Consortium. This has meant that the Mercator developers have been able to specify extensions to the X software onto which screen-reader-like adaptations can be hooked. Those extensions are part of the standard release of X. In other words, the hooks are irrelevant to most X users and not used by most X software developers. However, the creator of an adaptation can use those hooks to make other software accessible.

Given this level of access to the system, the developers of Mercator have been able to build a well structured interface. Access is based on a hierarchical structure imposed on the interface components. In this way the spatial layout of visual components is irrelevant and replaced by a logical structure. "Natural" non-speech sounds are used to give contextual information to the user. For instance, on entering a field into which text can be typed the user hears a typewriter noise. Synthetic speech is used to read out textual information. Mynatt and Weber (1994) describes the approach taken in Mercator, and contrasts it with that of GUIB.

VISUALLY IMPAIRED USERS

As often seems to be the case within information technology, the needs of visually impaired (i.e. partially sighted) users seems to have come further down the list of priorities. To some extent the basic problem is the same: the lack of bandwidth - but in the visual channel.

In addition to Outspoken, Berkeley Systems produced a screen enlarger for the Macintosh, called Inlarge. Shortly afterwards Apple bought the rights to a simplified version they called CloseView and for a time it was shipped as a standard part of the Macintosh system.

The most concerted effort to address the needs of visually impaired GUI users was the Unwindows Project, which was funded by the NSF (Kline and Glinert, 1995). This adaptation of X-Windows performs two functions of particular help to those with restricted vision: selective magnification of areas of the screen so that the contents can be seen comfortably, and keeping track of the location of the mouse pointer.

CONCLUSIONS

All of the above developments have followed the same progression in that a visual GUI has been developed and then at some later time someone else has thought about the needs of users with visual disabilities and some form of adaptation has been added on to the existing interface. To some extent the success of the adaptation has depended on the level of access to the existing visual software. Microsoft's Windows is the most used GUI and it is with the new release of this software, Windows 95, that possibly the greatest innovation will occur.

Having been seen as an adversary by many visually disabled people, Microsoft appears to have gone through a dramatic change of heart and is expending effort on ensuring that Windows 95 will be adaptable to the needs of visually disabled users. Right from its first release, Windows 95 will include an off-screen model which software developers will be able to use as the basis of non-visual representations of the screen.

This paper is about history, so it is probably inappropriate to speculate in it too much about the future. There are a range of GUI adaptations available and the change in attitude and software at Microsoft will probably prove to be significant. So it might be tempting to believe that the GUI problem is solved. However, I would caution that we should not get carried away yet.

GUIs are effective because they make the most of this capacity of sight and the problem of adapting them is due mainly to the fact that the other senses simply do not have that capacity. The limitation of the current adaptations is that they tend to use just one of the non-visual channels of communication.

In most cases they rely almost entirely on speech communication - supplemented to some extent by non-speech sounds. It seems unlikely that such a restricted channel can ever be a substitute for the richness of vision.

It is a shame that the original principle of the GUIB Project of using as many channels as possible was not followed through in terms of the commercial products which emerged. But on a positive note, Microsoft's approach of not supplying adaptations but providing the facilities whereby third parties can develop them might bear fruit. Assuming the off-screen model is sufficiently rich, multi-modal adaptations will be developed and I would expect it to be these which will be most successful and usable.

Manufacturers

References