Z39.50 - The Empire Strikes Back

6th December 2000

Introduction

In the first plenary session of the December 2000 ZIG meeting in Washington DC, three people - John Kunze, Poul Henrik Jørgensen and Matthew Dovey - shared ``provocative presentations'' which, while differing in content, were united in a perception that Z39.50 as it currently stands is not very useful.

This presentation is reverse-engineered from my impromptu rebuttal, in which I emphasised the strengths of Z39.50 as it stands today, and proposed a less radical route towards making it ``friendlier'' than those favoured by John, Poul Henrik and Matthew. Ray Denenberg said that he would put all the presentations up on the Maintenance Agency's web site, which is why I have created this document.

I've called it The Empire Strikes Back because I find myself cast against type as the reactionary who supports the Old Order Of Things. The evening before these presentations took place, Les Wibberley remarked to me that it's a continual failure of bright people coming into new situations that they tend to discount everything that's come before them, with all its accumulated wisdom. I found it ironic that in DC, it was John Kunze - a founding father of the ZIG - who wanted to throw everything away and start again; while I - a relative newcomer - want to keep more or less what we have now.

Unfortunately, in using this title, I am casting the existing order of things as an ``Evil Empire''; someone suggested that a better title might be ``The Return of the Jedi''!

Anyway, here it is.

Stated Strengths of Z39.50

The three previous presentations had little good to say about Z39.50, but what did come across is that there is widespread appreciation for the culture of abstraction in the Z39.50 community. Specifically, the separation of access points for searching from the physical data elements in which they search is seen as a very powerful tool (indeed the primary motivation of the Z+SQL project seemed to be a desire to use these abstract access points for SQL searches.)

This sort of abstraction is of course unrelated to the use of the Z39.50 protocol, and there is no real reason why its use should be unique to the Z39.50 community. However, for historical reasons, this does seem to be the case. With initiatives such as the Dublin Core, and more recently the RDF, the world seems gradually to be catching up with abstraction techniques that have been common in the Z39.50 world for some time.

Unstated Strengths of Z39.50

In addition to this, there are numerous additional strengths that were either not mentioned at all or glossed over in the other presentations. They include but are not limited to:

• Complex data. Binary data, such as images (JPEG, GIF, etc.) can be represented either as EXTERNAL records, or embedded in GRS-1 records. Complex structured data such as bibliographic holdings and the CIMI Profile's Multiple-Rendition Image also find natural representations in GRS-1. While the putative use of XML as a record syntax addresses some of these issues, it has problems with binary data.

• Complex searches. The Type-1 query allows searches for such non-textual terms as ranges, polygons, musical tunes and Les Wibberly's ten-thousand-node chemical formulae.

• Complex semantics. Z39.50 has evolved to meet the needs of systems which need to do more than simple search and retrieval: such complex semantics are provided by facilities such as Explain, Access Control, Resource Control and of course Extended Services. While some of these facilities could be approximated using (for example) an HTTP transport layer, some of them are fundamentally impossible across a medium so tightly tied to the request-response paradigm.

• It is here today. Most important of all, Z39.50 exists in multiple implementation which work well and interoperate correctly. Encodings for complex data have been designed, tested and deployed. Implementations have been tested, debugged, and do the things described above. In contrast, the suggested alternatives to Z39.50 as we know it today will need to go through a multi-year specification/implementation/testing/debugging/release cycle before they can be used to do real work.

Things We Must Change

Despite these very real strengths, no-one - least of all me - is suggesting that Z39.50 as we know it today is perfect. There are obviously problems, as evidenced by its relatively low take-up, at least in comparison to HTTP's ubiquity.

However, most of these problems are somewhat superficial: they are problems of communication rather than of substance. The world perceives Z39.50 as being more complicated to use than it really is. To address this problem, several things should be done:

• Lose the OSI terminology. Everyone agrees that in these days of TCP/IP's dominance as a transport layer, the standard document is anachronistic in using old OSI terms such as ``target'' and ``origin'' rather than the more familiar ``client'' and ``server''. This will be remedied in the forthcoming reaffirmation of the standard.

• Restructure the Z39.50 standard document. A more fundamental restructuring of the document would split the current service definitions and ASN.1 each into two sections: one for ``core'' services (Init, Search and Present) and one for ``periphery'' services (basically everything else.) This would significantly reduce the amount of reading that newcomers feel they need to do before they can understand and implement.

• Provide better tools. There are several existing toolkits for building Z39.50 applications, some of them excellent. However, there is a bias towards C, which is not a popular application-building language any more. We need to make tools available - preferably free - in other, higher-level languages such as Perl, Java and Python. These must implement well-defined, ZIG-endorsed APIs so that application writers can switch between toolkits at will.

• Publicise the tools we do have. The existing and new tools must be actively promoted. One easy step would be to provide a standard, centralised repository of the freely-available software on the Maintenance Agency web site.

Mistakes To Avoid

What we must avoid more than anything else is reinventing wheels, which is what W3C are doing. All of their work on XML, RDF, SOAP, XML Schemas, the XML Query etc. could be harshly but not unfairly characterised as attempts to reinvent in an HTTP environment functionality that Z39.50 already has - namely GRS-1, the Attribute Architecture, ASN.1/BER, schemas and the Type-1 Query respectively.

Further, Z39.50 as we have it today provides functionality which the ``Web Mafia'' are not even attempting to duplicate. These areas include:

• The full richness of GRS.1: metadata, variants, etc.
• Asynchronous server-initiated requests such as Access Control.

Conclusion

We know that we need an editorial reworking of standard. We also need to make and promote more and better tools that work with more fashionable languages and development environments.

However, we won't win the web people over by watering down what we have, because there are plenty of simple mechanisms out there already - not least the web itself, which is just perfect for simple searching and retrieval.

If the web is like a Volkswagen Beetle - small, cheap, easy to use - then Z39.50 is like a Harrier Jump Jet - big and powerful, and necessarily somewhat harder to use. For many purposes - nipping down the road to do some shopping, taking the kids to school - a Beetle is an excellent solution and a Harrier would be ridiculous overkill; likewise, the web is an excellent solution to many information needs. However, for people who need to fly, a Beetle with aero-engines attached with string is a dangerous hack. These people need Harriers.

In the same way, people who have real, complex, requirements for searching, retrieval and asynchronous semantics need Z39.50.