Short bio: Computer Scientist, FOSS supporter (read more)
Tux Machines (TM)-specific
X was originally created in the mid-80s by a research group from MIT. Its goal was to create a windowing system quite unlike any that had been conceived before. Thus X's design differs greatly from that of other windowing systems, having designed-in support for many elements which are unique features which in fact are nowadays often being hacked into other windowing systems.
More often than not, such attempts are kludgy and don't work well, because they lack the ground-up desig X offers.
X went through a number of iterations because the original releases were not under a copyleft license. Every Unix developer created his own version, usually only modifying small parts, resulting in divergence into many incompatible versions, most of which fell under proprietary licences.
As a result, a standards body was eventually created to oversee the development of X. This body, known as the X Consortium, includes amongt its members IBM, Hewlett Packard and even Microsoft.
The X server has two important functions. Firstly, it speaks to the hardware; this means the X server needs to contain the driver for your graphics card, mouse, keyboard etc. Secondly, it speaks to X clients (every X program, from xterm to OpenOffice.org is an X client). Thus no X client ever talks directly to the hardware.
The most common channel is Unix Domain Sockets (UDS, a very fast mechanism for interprocess communication on Unix) which provides the highest speeds for local usage (for example where the X server and X clients are on the same machine). However it can also run over several network protocols, such as TCP/IP, allowing you to use your local X server to run a program on a distant machine over the Internet.
Luckily, working directly with the X protocol is seldom needed because X also provides xlib. Xlib is essentially a library of standard X tasks, such as basic drawing primitives and event handling. Xlib is written in C (with wrappers to many languages) and it in turn speaks to the X protocol for you. Xlib takes care of the low-level detail part of using the X protocol, such as establishing a connection over the appropriate channel and talking to the server.
Today the two most important widget sets in the Linux world are GTK and QT respectively. Their importance is greatly enhanced by the fact that these are the two toolkits on which the Gnome and KDE desktops are respectively built. Many other widget sets exist, and although none are as feature-rich as GTK or QT, they are still often used.
The two most important desktop environments today are of course KDE and Gnome, as most new Linux applications are built for one or the other. Currently KDE and Gnome basically match each other for features and which one a user prefers tend to be a matter of taste rather than a technical decision. Almost all Linux users use one of these two. Old-time Unix users and programmers often shun them however, preferring minimalist desktops.