This is the first of ten pieces which follow on from the initial overview of this subject.

International standardisation has a problem: many people outside the process (and even a few inside it) do not understand, at a basic level, what it is. Defining “standardisation” is easy enough – standardisation is essentially just agreeing on a specification; the far trickier concept is “international”. Let us look at some things which do not define international standardisation:

• It does not mean that people from many nationalities are involved
• It does not mean that particular problems of localisation are particularly attended to
• It does not mean the standard has a good chance of applying anywhere on the planet

Of course all of these things can be, and nearly always are, characteristics of International standardisation; but they do not define it. No, what defines international standardisation is that the agreements made are, literally, inter-national – or, in other words, between nations. Not between individuals, or lobby groups, or movements, or government departments, or corporations – but nations. This is the defining characteristic of international standardisation.

It is a respect for this international essence which will guide much of what follows in these pieces. It accounts for the complexity of the process, and it explains its value. The international aspect is, I argue, the guiding principle on which the mechanisms of JTC 1 should be built, and by which the activities of JTC 1 should be judged.

Has International Standardisation a Future?

Some have argued that International Standardisation is not relevant to the modern world, and particularly to the modern world of ICT standardisation. Bob Sutor (VP of Standards, IBM) has written a number of weblog entries reacting to the 2008 standardisation of ICO/IEC 29500. He speculates thus:

[W]hile [ISO and IEC have] created thousands of standards for safety, mining, agriculture, and other areas, perhaps people are now shifting away from thinking that these groups should have anything to do with IT and interoperability standards. […] With the actions of the ISO and IEC, I think people have every reason to think that way. I feel that way.

And suggests,

I think people need to remember that important, and sometimes more important, standards work is taking place in standards groups like the W3C, OASIS, IEEE, OMG, and the OAGI. That is, in my opinion, the ultimate stamp of quality and acceptance need not be from the ISO or one of the other I** organizations.

Is Sutor’s feeling (or the feelings of the “people” he mentions) justified? Are ISO and IEC ICT standards irrelevant today as compared to those from vendor consortia like those Sutor mentions? Against this, the JTC 1 Directives themselves assert some benefits of international standardisation

International Standards (IS) issued by JTC 1 are considered the most authoritative standards on IT. […] ISs […] are recognised throughout the world, and in many countries constitute the technical regulatory basis for public procurement of IT goods and services. The transposition of a specification into an IS […] makes it eligible for such procurement, and hence widens the market recognition of such a specification.

The argument is however rather smug and circular, and amounts to “international standards are authoritative and that leads to their market impact; this impact makes them authoritative”. Starting from fundamentals, five more convincing arguments may, I think, be made.

1. Diversity as a Hedge

The principal alternative to having a strictly international organisation set ICT standards is to have a vendor-led consortium doing it. Whatever the relative merits of International vs. consortia standards, I would argue it is risky to argue for a world in which International standardisation has ceased to exist and we only have one type of standards body.

Over the last decade we have witnessed movement of experts back and forth between the consortium and International Standards world. It is presumptuous to declare standards could only be made one way, and to construct a world around that presumption. Those who argue against international standardisation itself are in effect declaring they are so sure of its worthlessness that it must be actively eliminated, presumably by dismantling the national standards bodies of all nations and dissolving the international standards frameworks in which they participate, including ISO, IEC, the UN bodies, and ITU-T.

2. Stability

Standards consortia are essentially commercially-based entities relying for their continued existence on the membership fees of their (mainly commercial) members. As such they are subject to the vagaries of the market and the collective whim of their membership. In recent years the market conditions in which consortia exist have become more difficult, and there is every reason to believe, with the recent difficulties in the global economy, that this situation will not improve. Indeed recent events have rather served to demonstrate that the apparently mightiest commercial entities are vulnerable compared to governments – the true last resort of stability. We can have no confidence today in the continuing existence of any particular ICT consortium in the short, let alone the medium, term.

The de jure standards organisations have the rock-like backing of governments supporting national standards bodies. This is a necessary and appropriate stability, considering that these de jure bodies are responsible for the stewardship of over 17,000 published standards (in ISO). I have heard no convincing proposals for what might happen to these published standards if the de jure bodies were dismantled.

3. A voice for governments

There is much talk of the benefits of “openness” to the standards world today, yet ironically many of the proposed non-international alternatives to JTC 1 that have been proposed (usually based on vendor-led consortia) are, from the perspective of a national government, closed. How does it work if the country (the USA, say) wants a voice in a consortium body? What does it mean for vendors established in that nation if their view conflicts with “national policy”? The de jure bodies, with their – partly necessary – elaborate bureaucracy, are designed to channel and mediate such national positions effectively and provide an “open” forum for the expression of nations’ voices. Individuals and corporations may feel disenfranchised by this, but that rather misses the point … this is international standardisation.

4. Wider participation and dissemination

Another clear benefit of the international standardisation mechanisms is their sheer size. With 83 participating nations (in JTC 1), each with their own collections of committees, the number of experts that may be called on is impressive, many of them bring distinctive and valuable requirements and expertise to the table. The recent standardisation of ISO/IEC 29500, for example, drew on the expertise of well over a thousand contributors form a wide variety of nations. It is hard to imagine non-international structures ever achieving similar levels of concentration of thought power.

Standards that are made internationally also have greater reach through being transposed into national standards by participating nations and translated into the native language(s) if necessary.

5. A bulwark against corporatism

As was observed above, nations are one of the few types of entity that can be relied on to provide better stability than global corporations. By the same token, nations are the only entities left on this planet with sufficient power to resist any untoward behaviour on the part of those corporations; nations collectively can, and frequently do, arrive at conclusions which dismay corporations. With the voices of nations removed from standardisation, there would be no bar to complete corporate dominance of the standardisation space.

It is vital that governments are allowed to participate in standardisation, since governments have (or should have) a very different kind of compact with their citizens than corporations have with their consumers. The guiding principle of corporate activity is profit; governments are in a position to take a longer-term and socially informed view of national interest. It is we, the users of ICT standards, who have much to lose if this dimension becomes excluded from the ICT standards world.

There is some inaccurate information swirling around on the web about the maintenance of ISO/IEC 26300:2006 – Open Document Format for Office Applications (OpenDocument) v1.0.

For those following the story of document format standardisation, this blog entry sets out the current situation ahead of the upcoming JTC 1 plenary in Nara, Japan, where this very topic is likely to be discussed and, one hopes, get debugged.

Background

The diagram above illustrates the current and planned major variants of the ODF standard.

The topmost is the OASIS standard 1.0, published by OASIS afters its approval in May 2005.

This OASIS standard was submitted by OASIS to JTC 1 for PAS transposition in October 2005. It passed its ballot with no dissent in May 2006, although a number of countries requested substantive fixes and improvements.

Because there had been no negative votes (only approves and abstention) in the ballot, the ballot resolution meeting (BRM) for the new standard was cancelled. (The UK objected to this decision at the May 2006 SC 34 plenary meeting in Seoul.)

Based on the comments from NBs, some substantive fixes and improvements were duly made to ODF, and ITTF incorporated these into the text of ISO/IEC 26300:2006, published in November 2006.

An equivalent text, an OASIS Committee Specification (not a standard, N.B.) called “OpenDocument v1.0 (Second Edition)”, had been published by OASIS in July 2006.

OASIS subsequently authored and published a new OASIS standard, ODF 1.1. This was published three months after ISO 26300:2006, i.e. in February 2007. OASIS did not seek cooperation in this from any part of ISO/IEC, nor did them submit the revised specification to JTC 1.

OASIS then began work on ODF 1.2, again without any ISO/IEC involvement.

In July 2008 the co-chair of the OASIS ODF TC announced in a blog entry: “[n]o one supports ODF 1.0 today. All of the major vendors have moved on to ODF 1.1, and will be moving on to ODF 1.2 soon.”

Throughout 2007 Japan, who were translating ISO/IEC 26300 into Japanese, fed reports of defects to OASIS via an OASIS mailing list. A formal set of Defect Reports was submitted by the Japanese National Body in December 2007 and circulated to SC 34 members and liaisons (including OASIS). The JTC 1 Directives state that the Project Editor must respond to a Defect Report for a JTC 1 standard within two months. SC 34 received no response until August 2008, when it was informed by the OASIS ODF TC that a register of errata in the OASIS standard had been published.

OASIS have produced errata document which apply corrections for some of the defects that have been reported. Note however that OASIS cannot amend the text which is the basis of ISO/IEC 26300, as this text has only the status of “Committee Specification” within OASIS. Hence they propose amending the defective OASIS 1.0 (“1st Ed”) Standard, creating a new fork of the ODF specification. SC 34 are expected to cross-apply these fixes to their corresponding locations within ISO/IEC 26300.

It is unclear whether the reported defects which also apply to ODF 1.1 are to be applied in any way.

Communications from OASIS make it clear that OASIS believes it has entered into an agreement with JTC 1 which allows it to maintain ISO/IEC 26300 in a way which exempts it from the maintenance provisions of the JTC 1 Directives.

Problems

OASIS’s continually restated stated intention in its communications with JTC 1 is to prevent divergence of ODF versions. This goal has clearly not been realised, with a proliferation of versions of ODF inside OASIS and pronounced marketplace confusion.

For example, it should be of concern to JTC 1 members that the OpenOffice.org product is promoted as supporting “features of the upcoming version 1.2 of the ISO standard OpenDocument Format (ODF)”.

OASIS’s continually restated intention in its communications with JTC 1 is to maintain a collaborative relationship. However there has not always been evidence of collaboration. Input from the ISO/IEC members has not been sought. Where input has been provided, it has sometimes been met with delay and dismissiveness.

The agreement that JTC 1 has reached with OASIS appears to be being operated in a way which breaches the JTC 1 Directives. The relevant portions of the Directives are given below (all emphasis mine):

Maintenance for a transposed PAS is also negotiated in the Explanatory Report. JTC 1's intention for maintenance is to avoid any divergence between the current JTC 1 revision of a transposed PAS and the current revision of the original specification published by the PAS submitter. Therefore, the Explanatory Report should contain a description of how the submitting organisation will work cooperatively with JTC 1 on maintenance of the standard. While JTC 1 is responsible for maintenance of the standard, this does not mean that JTC 1 itself must perform the maintenance function. JTC 1 may negotiate with the submitter the option of maintenance handled by the submitter as long as there is provision for participation of JTC 1 experts, i.e. the submitter's group responsible for maintenance is designated as the JTC 1 maintenance group. (Directives, 14.4.2)

For the maintenance of an International Standard of whatever origin normal JTC 1 rules apply. Such rules distinguish between correction of defects and revisions of or amendments to existing Standards. Note: The JTC 1 rules for maintenance are found in clause 15 of the JTC 1 Directives. For the correction of defects, JTC 1 provides for the installation of an editing group. Active participation of the submitter in such an editing group is expected and strongly encouraged. Depending on the degree of openness of the PAS submitter, JTC 1 will determine its specific approach. (Directives, M6.1.5)

Therefore it is clear that while maintenance may (in the lax wording of the Directives) be “handled” by the submitter, it is not possible for the submitter to exempt themselves from normal JTC 1 rules, as “for the maintenance of an International Standard of whatever origin normal JTC 1 rules apply”. From this it follows that a submitter’s “handling” of maintenance is limited, and that the decision-making procedures and time periods specified by the JTC 1 Directives must apply.

Remedies?

Obviously this is all an enormous mess and while it is tempting to blame lawyerly over-cleverness on the part of OASIS, or insufficient alertness on the part of JTC 1, in negotiating their so-called maintenance agreement, the true culprit is, in my view, the JTC 1 Directives – such an impenetrable document has, evidently, led to a completely different understanding of the situation from the several parties involved. This procedural mishap is, I argue, further evidence of the need to scrap and re-write the JTC 1 Directives as a short, clear and professionally drafted document. Already this year we have seen that when tough questions get asked, the Directives are not fit for purpose; we are seeing the same thing again now.

The immediate problem faced is, however, the future of ODF in JTC 1. This is not a matter for SC 34, or for the ODF TC (both of which groups are full of excellent  technical experts wanting nothing more that to produce good standards) – this is something that must be resolved at a higher level between JTC 1 and OASIS. In the usual way of things, the developers are being hampered by the management.

The essence of the problem is that a central principle is being missed now: that only a standard that has a truly international dimension to its control should benefit from the ISO, IEC or JTC 1 “brand”. Some immediate remedies might include some mix of the following:

• Since there seems to be general agreement that ISO/IEC 26300 is an obsolete version of ODF, perhaps it should be withdrawn as an IS – maybe in parallel with a PAS submission of ODF 1.1. That would at least give the world an IS that was widely used and a veil could be drawn over the 1.0 standardisation mess.
• SC 34 has already stated it is open to suggestions how future maintenance should be arranged in a genuinely collaborative manner. Patrick Durusau (the ODF editor) has drafted a proposed agreement in that spirit. Also, OASIS might well have a thing or two to learn by looking at how Ecma has managed to enter into a collaborative arrangement for the maintenance of ISO/IEC 29500 within JTC 1.
• The immediate defects in ISO/IEC 26300:2006 could be resolved by the formation of an editing group in SC 34. Indeed OASIS itself seemed to expect this in the explanatory report which accompanied their initial PAS submission which stated: “OASIS requests that any corrections of defects or errata from the JTC1 process be re-presented to the OASIS Technical Committee.” Per the Directives, OASIS TC members should be encouraged to participate in any such group.

Ultimately, it is for the nations participating in JTC 1 to decide how this matter can be resolved. The current situation sells-out the nations by allowing their brand (“international”) to be perpetuated in a process from which they are effectively excluded. This is “standardisation by corporation” through the back door. Whatever is decided, this must not go on.

This year's must-do geek photographer activity seems to be the production of HDR (High Dynamic Range) photos. Here is my first attempt ...

For any of you not familiar with the technique, the theory is that several  images of the same scene, which are identical except for exposure, can be cleverly combined by computer to produce a high dynamic range image. Because one image will have exposed highlights correctly, and one shadows, it is possible to have all parts of the image exposed correctly even in a high-contrast scene, such as one containing bright sky and shadows.

So much for the theory. In practical terms what you probably need (or at least what I have used is):

• A camera that does automatic bracketing with sufficient exposure differential (my Nikon D50 will take a three-image bracketed set at -2, 0 and +2 stops)
• A heavy tripod - each scene must be the same so the camera needs to be held firmly in place
• A remote shutter release - optional, but helps greatly to keep the camera un-moved
• Software - I've used Photomatix.

That's the basics, for an in-depth tutorial and a glimpse of what a master can do with HDR, check out Stuck in Customs.

So anyway, here is an autumn scene at Wandlebury, taken with the auto bracketing function. Note the rather bland "correctly exposed" image (centre), the too dark underexposed one to its left, and the more vivid over-exposed image on the right (with a blown-out white sky, actually it was blue).

All of these images were taken with a Nikon D50 and Nikkor 18-200 VR lens at 32mm and F11: only the exposure time varied. Before combination the images were corrected for lens abberations using DxO Optics Pro (processing from RAW). Here is the result of combining the three images using the "tone mapping" feature (and default settings) of Photomatix:

and a little further on down the path ...

yielding …

To me HDR images don't look much like photographs - they look much more like reality...

2008 has been an exciting year for document format standards. 2009 will, I predict, be rather more boring.

This at least is the conclusion I reached after attending the recent DII workshop organised by Microsoft – and if I say the event was boring I merely mean that we can confidently expect document formats to stop being the at the centre of a spectator sport, and start returning to the land of techies and standards wonks. Boring, but reassuringly so; for while the more slashdotty spectators may prefer the ya-boo exchanges that characterised 2008, for us techies and standards wonks, boring is good – even … exciting.

Boring includes discussion of such topics as:

• The effect of hyphenation dictionaries and justification algorithms on line breaking, and the impact of these considerations on achieving reproducible documents across implementations
• How to decide what the chief document archetypes were for spreadsheets, word-processing documents and presentations
• The distinction between an erratum and an amendment for an IEC/ISO standard
• How to assemble and administer a collection of representative documents for assessing implementation conformance
• How to validate the semantic constraints inherent in an OOXML document
• The trade-off between format-specific and generic document APIs
• How to facilitate server-side document generation
• The trade-off between user convenience and standards adherence
• The quirks of string sharing in Excel
• How to document the implementation decisions an application makes which imposing further constraints on the underlying XML
• How to re-purpose legacy Authorware training materials into OOXML

All good solid stuff, laying the groundwork for the people who really matter in this process (and who perhaps have too often been overlooked) – the end users. Doug Mahugh has a further write-up and links to the presentations on his blog.

Granted, a few eyebrows were raising during one presentation (which has not appeared among the others) which gave a startling frank overview of the challenges Microsoft are anticipating in implementing ISO/IEC 29500, from sucky performance in the deserialision code in PowerPoint, to dumb mistakes in Ecma 376, to coping with the fact that under certain circumstances Office 2007 emits content which is invalid against the Ecma 376 schemas.

I found this last revelation truly heartening – Microsoft did not need to make it, and to date (so far as I am aware) nobody has “caught” Office 2007 emitting invalid XML content. Yet here was MS ’fessing up and asking about ways they could stop it happening in future. All software companies (not just Microsoft) need to have a plain-dealing up-front approach to publicising problems. Such an approach has benefitted the security landscape and will have big benefits for document processing and conformance (and yes, for interoperability too). It is good to see the seeds of such a mature approach – I look forward to seeing Microsoft make this information public soon, and to something equivalent starting up for non-Microsoft ODF implementers too, bearing in mind that (with apologies to Alexander Pope):

Whoever thinks a bug-free app to see,
Thinks what n'er was, nor is, nor e'er shall be.

So, Talking of Bugs and ODF …

ODF Table Test

I had prepared a moderately hard table rendering test to take to Redmond, reasoning that table rendering is a fair indicator of the state of a layout engine beyond a basic “text and headings” level. To create this test document at home perform the following steps:

1. Create a 5x5 Table
2. Number the cells starting at the top left and moving left-to-right, top-to-bottom until you reach number 25 at the bottom right
3. Merge consecutive cells to achieve the result below (note I have also coloured the merged cells to make it easier to see what has happened).

Et voila, a table rendering test. Here is that table displayed in OpenOffice 2.4 (click to enlarge):

Now, let us see how OpenOffice’s version of the table opens with the SP2 beta for Word 2007:

Good – on the face of it, a mini triumph for interoperability. For comparison, I also tried to open the document with Google Docs:

Hmm  – notice the different rendering here. Most obviously, the yellow cell which combines the original cells 4, 9 and 14 does not span downward, whereas that is what wanted to achieve when we created the table.

Looking at the ODF source, everything appears to be in order. The top-most spanned cell is marked-up as follows

<table:table-cell table:style-name="Table1.A1"
    table:number-rows-spanned="3" office:value-type="string">
  <text:p text:style-name="Table_20_Contents">4 9 14</text:p>
</table:table-cell>

The number-rows-spanned="3" attribute specified the row-spanning correctly, and the spanned-into cells (not shown) are properly marked-up with <table:covered-table-cell/> elements as the ODF spec suggests. (Interesting note: at no point is the ODF spec explicit that row spanning operations apply downwards – I have come across XML table models – Arbortext’s for example - which specify that spans apply upwards, and it is theoretically open for an ODF implementation to chose to do that too. So much for interoperability!)

So I think here we can reasonably point the finger at Google Docs and say that its table renderer is faulty – these guys need to catch up with OpenOffice and Microsoft.

Curiously, opening this test file with an early version of OpenOffice.org (1.1.5) gives the same rendering error as today’s Google docs:

And so we have seen a minor failure of interoperability. Of course it might not be so minor if these documents contained more important information (financial or medical data, e.g.) and not just pretty colours. This test however, just scratches the surface – for a more thoroughgoing examination of the poor state of ODF interoperability, readers can turn to the recent study by Shah and Kesan (and no, OOXML does not emerge hugely better from this either).

Looking Forward

Achieving interoperability appears to be the new focus for both the developers and standardisers working on document formats. For ODF, OASIS has the new Open Document Format Interoperability and Conformance (OIC) TC to advance work in this area. Microsoft is not currently represented here, and it is to be hoped they might soon overcome their shyness and participate. After all, when Office 2007 SP2 ships, Microsoft Office will quickly become the predominant ODF implementation, and it is important everybody works together to ensure they improve conformance and interoperability, and that where the ODF specification is insufficient for this, feedback is returned to ODF’s custodians.

Microsoft too are evidently thinking about interoperability and several presentations at the DII workshop were concerned with work to build a repository of representative Office documents to provide input into conformance and interoperability testing processes. Such an initiative is useful, though ideally it should take place under the aegis of a standards committee (e.g.  OASIS or SC 34 / WG 4), to parallel the activities taking place for ODF.

There was indeed, plenty of corridor discussion about how the future standards arrangements for ODF and OOXML might be best organized. I was particularly pleased to meet Dennis Hamilton (aka Orcmid), a member of the OASIS ODF TC and secretary to the OIC TC there – and we had plenty of constructive discussions about the how the current impasse in ODF maintenance might best be negotiated. However, the immediate solution to that particular problem now lies above the reach of mere committee members like us; it is between the lawyers and officials of OASIS and JTC 1.

R&R

After the workshop finished, there was time before my midnight flight for some R&R. Doug Mahugh was good enough to indulge my confession to being a Frasier fan and give me a tour of Seattle (where he grew up). It was good too to take a break from document formats and discuss less controversial topics like the US election, the identity of Mini-Microsoft, and Kirk/Spock porn!

Me being shown Seattle culture (photo: Doug Mahugh)

[UPDATE: Jesper Lund Stocholm has also just blogged on the topic of ODF support in MS Office — highly recommended.]

[UPDATE 2: wifely perspective.]

… Big …

For those who missed it, last week ISO announced the appointment of a new Secretary General from 2009, Mr Rob Steele. By reputation Mr Steele is something of a moderniser, widely held to have had a very positive impact at Standards New Zealand in his role there as CEO. His written opinions too speak of somebody engaged with the challenges faced within international standardisation today. It is to hoped that from his lofty perch within ISO he will also be a force for the good in facilitating the necessary change JTC 1 so badly needs.

… and Small …

By zooming down from the Secretary General through three levels of subcommittee we can end up at the shiny new working group of SC 34’s WG 4, established for the maintenance of IEC/ISO 29500 (aka OOXML). I was delighted to learn recently that Denmark’s excellent Jesper Lund Stocholm has been appointed as Dansk Standard’s technical expert for this WG (kudos to CIBER for supporting this work). It is great to see this kind of independent expertise coalescing around this important new standard. Excitement is already mounting about this group’s first meeting scheduled to take place in Okinawa in January 2009, alongside WG 5 which is concentrating on document interop. Did I really hear it right that one country is considering sending a delegation of 6 experts to Okinawa?

Jesper Lund Stocholm