There has been quite a lot of hubbub recently about ODF conformance, in particular about how conformance to the forthcoming ODF 1.2 specification should be defined.

A New Conformance Clause

Earlier versions of ODF (including ISO/IEC 26300) already defined conformance - it was simply a question of obeying the schema. So in ODF 1.1, for example, we had this text:

Conforming applications [...] shall read documents that are valid against the OpenDocument schema if all foreign elements and attributes are removed before validation takes place [...] (1.5)

and that was the simple essence of ODF conformance.

This is now up for reconsideration. The impetus for altering the existing conformance criteria appears to have come from a change in OASIS's procedures, which now require that specifications have “a set of numbered conformance clauses”, a requirement which seems sensible enough.

However, the freshly-drafted proposal which the OASIS TC has been considering goes further than just introducing numbered clauses: it now defines two categories of conformance:

1. “Conforming OpenDocument Document” conformance
2. “Conforming OpenDocument Extended Document” conformance

as shorthand, we might like to characterise these as the “pure” and “buggered-up” versions of ODF respectively.

The difference is that the “pure” version now forbids the use of foreign elements and attributes (i.e. those not declared by the ODF schema), while the “buggered-up” version permits them.

Ructions

The proposal caused much debate. In support of the new conformance clause, IBM's Rob Weir described foreign elements (formerly so welcome in ODF) as proprietary extensions that are “evil” and as a “nuclear death ray gun”. Questioning the proposal, KOffice's Thomas Zander wrote that he was “worried that we are trying to remove a core feature that I depend on in both KOffice and Qt”. Meanwhile Microsoft's Doug Mahugh made a counter-proposal suggesting that ODF might adopt the Markup Compatibility and Extensibility mechanisms from ISO/IEC 29500 (OOXML).

Things came to a head in a 9-2-2 split vote last week which saw the new conformance text adopted in the new ODF committee specification by will of the majority. Following this there was some traffic in the blogosphere with IBM's Rob Weir commenting and Microsoft's Doug Mahugh counter-commenting on the vote and the circumstances surrounding it.

Shadow Play

What is to be made of all this? Maybe Sun, whose corporate memory still smarts from Microsoft's “extend and embrace” Java attempts, thinks this is a way to prevent a repeat of similar stunts for ODF. Or perhaps this is a way to carve out a niche for OpenOffice to enjoy “pure” status while competitor applications are relegated to the “buggered-up” bin. Maybe it is envisaged that governments might be encouraged to procure only systems that deal in “pure” ODF. Maybe foreign elements really are the harbinger of nuclear death.

Who knows?

Whatever the reasons behind the reasons, there is clearly an “absent presence" in all these discussions: Microsoft Office. And in particular the forthcoming Microsoft Office 2007 SP2 with its ODF support. It is never mentioned, except in an occasional nudge-nudge wink-wink sort of way.

This controvery is most bemusing. This is in part because the “Microsoft factor” appears not to be a factor anyway, since MS Office will (we are told) not use foreign elements for its ODF 1.1 support. But the main reason why this is bemusing is that this discussion (whether or not to permit foreign elements) is completely unreal. There seems to be an assumption that it matters – that conformance as defined in the ODF spec means something important when it comes to real users, real procurement, real development or real interoperability.

It doesn't mean anything real - and here's why...

Making an ODF-conformant Office Application

Let us consider the procurement rules of an imaginary country (Vulgaria, say). Let us further imagine that Vulgaria's government wants to standardize on using ODF for all its many departments. After many hours of meetings, and the expenditure of many Vulgarian Dollars on consultancy fees, the decision is finally made and an official draws up procurement rules to stipulate this:

Any office application software procured by the Government of Vulgaria must support ODF (ISO/IEC 26300), and must conform to the 'pure' conformance class defined in clause x.y of that Standard, reading and emitting only ODF documents that are so conformant".

Sorted, they think.

Now imagine a software company that has its eye on making a big sale of software licenses to Vulgaria. Unfortunately, its office application does not meet the ODF conformance criterion set out by the procurement officer. The marketing department is duly sad. But one day a bright young developer gets to hear of the problem and proposes a solution. He boldy proclaims “I can make our format ODF-conformant today!”, and proceeds to show how.

First he gets a template ODF document, like this:

<?xml version="1.0" encoding="UTF-8"?>
<office:document-content
xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0"
xmlns:text="urn:oasis:names:tc:opendocument:xmlns:text:1.0"
office:version="1.0">
<office:body>
<office:text>
<text:p></text:p>
</office:text>
</office:body>
</office:document-content>

This document (he points out) meets the “pure” conformance criteria. Our young hacker then does a curious thing: he takes an existing (non-ODF) file from their office software, BASE-64 encodes it, and inserts the resulting text string into the element in the template document.

<?xml version="1.0" encoding="UTF-8"?>
<office:document-content
xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0"
xmlns:text="urn:oasis:names:tc:opendocument:xmlns:text:1.0"
office:version="1.0">
<office:body>
<office:text>
<text:p><!-- several MBs of BASE-64 encoded content here --></text:p>
</office:text>
</office:body>
</office:document-content>

There, he proudly proclaims. All we need to do it to wrap our current documents with the ODF wrapper when we save, and unwrap when we load – I can have a fresh build for you tomorrow.

The rest of the story is not so happy: the software company makes the sale and the government of Vulgaria finds after installation that none of the files from it will interoperate with any other ODF files from other sources, despite the software company having met its procurement rules to the letter.

Far fetched?

Okay, that story makes an extreme example – but it neverthess illustrates the point. It is possible for a smart developer to represent pretty much anything as a “pure” ODF document; any differences and incompatibilities can ever-so-easily be shoehorned into conformant ODF documents. That some software deals only in such pure ODF means precisely zero in the real world of interoperability.

The central consideration here is that ODF conformance only ever was (and is only projected to be) stated in terms of XML, and XML is (in)famously “all syntax and no semantics”. The semantics of an ODF document (broadly, all the narrative text in the specification) play no part in conformance can remain unimplemented in a conformant processor. An ODF developer can safely use just the schema and never read much else. All those descriptions of element behaviour can be ignored for the purposes of achieving ODF conformance. [N.B. mistakes in this para corrected following comment from Rob Weir, below]

So my question is: what is the current debate on ODF conformance really about? It looks to me like mis-directed effort.

What ODF might usefully do is to look at the “application description” feature introduced into OOXML. This describes several types of applications, including a type called “full”. Such applications have “a semantic understanding of every feature within [their] conformance class”, and

“Semantic understanding” is to be interpreted that an application shall treat the information in Office Open XML documents in a manner consistent with the semantic definitions given in this Specification.

In other words, it is possible to specify in OOXML procurement that the processor should heed the narrative description within that Standard (not just the XML grammar). ODF currently lacks this. In my view if there is to be any connection between a definition of ODF conformance and the experience of users in the real world, then something like OOXML's “application description” feature is urgently needed. And it might be better done now, than hastily inserted during a JTC 1 BRM ...

“JTC 1 recognizes the timely response (JTC 1 N9398) from OASIS to the SC 34 liaison statement (SC34 N1095 […]), and thanks OASIS for the new draft errata to ODF 1.0. JTC 1 particularly welcomes OASIS's proposal to confer with JTC 1 and SC 34 to forge a genuine partnership for collaboratively handling the maintenance of ODF/IS 26300. JTC 1 requests SC 34 and OASIS to develop a document specifying the detailed operation of joint maintenance procedures, with a common goal of preparation of technically-equivalent documents, and taking into account the requirements and constraints of both standards bodies. SC 34 is requested to consider this document at its March 2009 plenary and report the results to JTC 1 following this meeting.”

(See the SC 34 chairman’s Business Plan, as presented in Nara, for
this and other interesting information.)

The prelude to this resolution is a sequence of exchanges between SC 34 and OASIS. Now, while highly selective leaking to unwitting and credulous sites may have succeeded in producing a fuss in the blogosphere (see, for example, groklaw's “The Microsoft-Stacked SC 34 Committee Makes a Move”) the truth is rather less sensational, and speaks more of parties of good will wanting to make progress, than of the crazed oppositional narrative of “MS vs the world” that the tinfoil brigade seems increasingly desperate to try to perpetuate. The liaison statement from SC 34 to OASIS out of Jeju was, of course, not leaked to/by groklaw because it did not fit with that crazed narrative. I don’t believe it is giving too much away to reveal its concluding words were: “SC 34 is open to suggestions as to how to reach a resolution of this issue that is mutually acceptable to OASIS and SC 34.”

The tinfoil hat wearers are desperate to construct a a narrative around ODF in which MS plays the villain; facts are getting in their way. (Photo credit: Rob Watkins. Licence.)

OASIS duly replied indicating in the course of their communication that they too were interested in such a mutually acceptable resolution, in particular for the maintenance issues (of errata and defects) that had arisen from the current unsatisfactory maintenance agreement.

And so it was that in Nara representatives of JTC 1, SC 34, OASIS and some of the commercial stakeholders in ODF worked hard and hammered out the text above, which was duly amended and blessed by the JTC 1 members (nations) – who are, ultimately, the decision makers in charge of international standardisation.

Reading the Runes

The first two sentences of the resolution set out the background. The third contains the meat:

“JTC1 requests SC34 and OASIS to develop a document specifying the detailed operation of joint maintenance procedures, with a common goal of preparation of technically-equivalent documents, and taking into account the requirements and constraints of both standards bodies.”

The three key phrases here are, I think, these:

• “joint maintenance procedures” – critically maintenance (in JTC 1 terms “maintenance” includes the following activities: revision, withdrawal, periodic review, correction of defects, amendment, and stabilization) will now be a joint activity, rather than one conducted exclusively in isolation.
• “technically-equivalent documents” – so, documents must be the same (apart from such non-technical things as cover pages). By keeping the OASIS and International Standard versions in step-lock with each other, marketplace confusion can be avoided by eliminating doubts about version differences
• “the requirements and constraints of both standards bodies” – OASIS and JTC 1 have different ways of doing things; some way will need to be found so that all concerns are properly met.

Now, I have no idea what the final maintenance agreement is going to look like. SC 34 people and OASIS are going to keep working hard over the next few months and it is anticipated these negotiations will culminate in a face-to-face summit to be held in Okinawa at the end of January 2009, to coincide with the meetings of WG 4 (dedicated to OOXML) and WG 5 (dedicated to document format interop, particularly ODF/OOXML). Any agreed text will ultimately need to be blessed by the two top-levels of the organisations … this is, after all, an agreement between JTC 1 and OASIS, and not between SC 34 and OASIS, or SC 34 and the ODF TC. Okinawa certainly looks like it is going to be the site of a vibrant meeting, with OOXML and ODF folks attending in numbers…

My personal hunch about the shape of the final maintenance arrangement is that it will be less like the one SC 34 arranged with Ecma, in which the Ecma TC was absorbed into a new Working Group, and something more akin to a parallel-running process, with mechanisms for exchanging information and synchronising key activities. But that is just my personal hunch.

Spreading the Love

Via Doug Mahugh, from Redmond, comes the happy announcement (even IBM’s Bob Sutor called it “excellent news”) that Microsoft will be participating in OASIS’s ODF Interoperability and Conformance TC (see Rob Weir’s post for background on this activity). This is really good to hear. With the release of Office 2007 SP2 Microsoft are suddenly going to find themselves stewards of by far the biggest installed user-base of ODF office applications, so it is vital for users they are part of the conversation developers and vendors need to be having about making their implementations interoperate.

From the uncertainty that marked the beginning of the year, these latest pieces of news are very positive indications of progress in the document format space. So much has been accomplished in 2008, and I have every confidence 2009 is going to see this positive progress continue …

There is some inaccurate information swirling around on the web about the maintenance of ISO/IEC 26300:2006 – Open Document Format for Office Applications (OpenDocument) v1.0.

For those following the story of document format standardisation, this blog entry sets out the current situation ahead of the upcoming JTC 1 plenary in Nara, Japan, where this very topic is likely to be discussed and, one hopes, get debugged.

Background

The diagram above illustrates the current and planned major variants of the ODF standard.

The topmost is the OASIS standard 1.0, published by OASIS afters its approval in May 2005.

This OASIS standard was submitted by OASIS to JTC 1 for PAS transposition in October 2005. It passed its ballot with no dissent in May 2006, although a number of countries requested substantive fixes and improvements.

Because there had been no negative votes (only approves and abstention) in the ballot, the ballot resolution meeting (BRM) for the new standard was cancelled. (The UK objected to this decision at the May 2006 SC 34 plenary meeting in Seoul.)

Based on the comments from NBs, some substantive fixes and improvements were duly made to ODF, and ITTF incorporated these into the text of ISO/IEC 26300:2006, published in November 2006.

An equivalent text, an OASIS Committee Specification (not a standard, N.B.) called “OpenDocument v1.0 (Second Edition)”, had been published by OASIS in July 2006.

OASIS subsequently authored and published a new OASIS standard, ODF 1.1. This was published three months after ISO 26300:2006, i.e. in February 2007. OASIS did not seek cooperation in this from any part of ISO/IEC, nor did them submit the revised specification to JTC 1.

OASIS then began work on ODF 1.2, again without any ISO/IEC involvement.

In July 2008 the co-chair of the OASIS ODF TC announced in a blog entry: “[n]o one supports ODF 1.0 today. All of the major vendors have moved on to ODF 1.1, and will be moving on to ODF 1.2 soon.”

Throughout 2007 Japan, who were translating ISO/IEC 26300 into Japanese, fed reports of defects to OASIS via an OASIS mailing list. A formal set of Defect Reports was submitted by the Japanese National Body in December 2007 and circulated to SC 34 members and liaisons (including OASIS). The JTC 1 Directives state that the Project Editor must respond to a Defect Report for a JTC 1 standard within two months. SC 34 received no response until August 2008, when it was informed by the OASIS ODF TC that a register of errata in the OASIS standard had been published.

OASIS have produced errata document which apply corrections for some of the defects that have been reported. Note however that OASIS cannot amend the text which is the basis of ISO/IEC 26300, as this text has only the status of “Committee Specification” within OASIS. Hence they propose amending the defective OASIS 1.0 (“1st Ed”) Standard, creating a new fork of the ODF specification. SC 34 are expected to cross-apply these fixes to their corresponding locations within ISO/IEC 26300.

It is unclear whether the reported defects which also apply to ODF 1.1 are to be applied in any way.

Communications from OASIS make it clear that OASIS believes it has entered into an agreement with JTC 1 which allows it to maintain ISO/IEC 26300 in a way which exempts it from the maintenance provisions of the JTC 1 Directives.

Problems

OASIS’s continually restated stated intention in its communications with JTC 1 is to prevent divergence of ODF versions. This goal has clearly not been realised, with a proliferation of versions of ODF inside OASIS and pronounced marketplace confusion.

For example, it should be of concern to JTC 1 members that the OpenOffice.org product is promoted as supporting “features of the upcoming version 1.2 of the ISO standard OpenDocument Format (ODF)”.

OASIS’s continually restated intention in its communications with JTC 1 is to maintain a collaborative relationship. However there has not always been evidence of collaboration. Input from the ISO/IEC members has not been sought. Where input has been provided, it has sometimes been met with delay and dismissiveness.

The agreement that JTC 1 has reached with OASIS appears to be being operated in a way which breaches the JTC 1 Directives. The relevant portions of the Directives are given below (all emphasis mine):

Maintenance for a transposed PAS is also negotiated in the Explanatory Report. JTC 1's intention for maintenance is to avoid any divergence between the current JTC 1 revision of a transposed PAS and the current revision of the original specification published by the PAS submitter. Therefore, the Explanatory Report should contain a description of how the submitting organisation will work cooperatively with JTC 1 on maintenance of the standard. While JTC 1 is responsible for maintenance of the standard, this does not mean that JTC 1 itself must perform the maintenance function. JTC 1 may negotiate with the submitter the option of maintenance handled by the submitter as long as there is provision for participation of JTC 1 experts, i.e. the submitter's group responsible for maintenance is designated as the JTC 1 maintenance group. (Directives, 14.4.2)

For the maintenance of an International Standard of whatever origin normal JTC 1 rules apply. Such rules distinguish between correction of defects and revisions of or amendments to existing Standards. Note: The JTC 1 rules for maintenance are found in clause 15 of the JTC 1 Directives. For the correction of defects, JTC 1 provides for the installation of an editing group. Active participation of the submitter in such an editing group is expected and strongly encouraged. Depending on the degree of openness of the PAS submitter, JTC 1 will determine its specific approach. (Directives, M6.1.5)

Therefore it is clear that while maintenance may (in the lax wording of the Directives) be “handled” by the submitter, it is not possible for the submitter to exempt themselves from normal JTC 1 rules, as “for the maintenance of an International Standard of whatever origin normal JTC 1 rules apply”. From this it follows that a submitter’s “handling” of maintenance is limited, and that the decision-making procedures and time periods specified by the JTC 1 Directives must apply.

Remedies?

Obviously this is all an enormous mess and while it is tempting to blame lawyerly over-cleverness on the part of OASIS, or insufficient alertness on the part of JTC 1, in negotiating their so-called maintenance agreement, the true culprit is, in my view, the JTC 1 Directives – such an impenetrable document has, evidently, led to a completely different understanding of the situation from the several parties involved. This procedural mishap is, I argue, further evidence of the need to scrap and re-write the JTC 1 Directives as a short, clear and professionally drafted document. Already this year we have seen that when tough questions get asked, the Directives are not fit for purpose; we are seeing the same thing again now.

The immediate problem faced is, however, the future of ODF in JTC 1. This is not a matter for SC 34, or for the ODF TC (both of which groups are full of excellent  technical experts wanting nothing more that to produce good standards) – this is something that must be resolved at a higher level between JTC 1 and OASIS. In the usual way of things, the developers are being hampered by the management.

The essence of the problem is that a central principle is being missed now: that only a standard that has a truly international dimension to its control should benefit from the ISO, IEC or JTC 1 “brand”. Some immediate remedies might include some mix of the following:

• Since there seems to be general agreement that ISO/IEC 26300 is an obsolete version of ODF, perhaps it should be withdrawn as an IS – maybe in parallel with a PAS submission of ODF 1.1. That would at least give the world an IS that was widely used and a veil could be drawn over the 1.0 standardisation mess.
• SC 34 has already stated it is open to suggestions how future maintenance should be arranged in a genuinely collaborative manner. Patrick Durusau (the ODF editor) has drafted a proposed agreement in that spirit. Also, OASIS might well have a thing or two to learn by looking at how Ecma has managed to enter into a collaborative arrangement for the maintenance of ISO/IEC 29500 within JTC 1.
• The immediate defects in ISO/IEC 26300:2006 could be resolved by the formation of an editing group in SC 34. Indeed OASIS itself seemed to expect this in the explanatory report which accompanied their initial PAS submission which stated: “OASIS requests that any corrections of defects or errata from the JTC1 process be re-presented to the OASIS Technical Committee.” Per the Directives, OASIS TC members should be encouraged to participate in any such group.

Ultimately, it is for the nations participating in JTC 1 to decide how this matter can be resolved. The current situation sells-out the nations by allowing their brand (“international”) to be perpetuated in a process from which they are effectively excluded. This is “standardisation by corporation” through the back door. Whatever is decided, this must not go on.