Notes on Document Conformance and Portability #3

by Alex Brown 10. May 2009 13:20

Now that the furore about Microsoft’s implementation of ODF spreadsheet formulas in Office SP2 has died down a little, it is perhaps worth taking a little time to have a calm look at some of the issues involved.

Clearly, this is an area where strong commercial interests are in play, not to mention an element of sometimes irrational zeal from those who consider themselves pro or anti (mostly anti) Microsoft.

One question is whether Microsoft did “The Right Thing” by users in choosing to implement formulas the way they did. This is certainly a fair question and one over which we can expect there to be some argument.

The fact is that Microsoft’s implementation decision means that, on the face of it, they have produced an implementation of ODF which does not interoperate with other available implementations. Thus IBM blogger Rob Weir can produce a simple (possibly simplistic) spreadsheet, “Maya’s Wedding Planner” and used it to illustrate, with helpful red boxes for the slow-witted, that Microsoft’s implementation is a “FAIL” attributable to “malice or incompetence”. For good measure he also takes a side-swipe at Sun for their non-interoperable implementation. In this view, interoperability aligning with IBM’s Symphony implementation is – unsurprisingly – presented as optimal (in fact, you can hear the sales pitch from IBM now: “well, Mr government procurement officer, looks like Sun and MS are not interoperable, you won’t want these other small-fry implementations, and Google’s web-based approach isn’t suitable – so looks like Symphony is the only choice …”)

Microsoft have argued back, of course, most strikingly in Doug Mahugh’s 1 + 2 = 1? blog posting, which appears to present some real problems with basic spreadsheet interoperability among ODF products using undocumented extensions. The MS argument is that practical ODF interoperability is a myth anyway, and so supporting it meaningfully is not possible (in fact, you can hear the sales pitch from MS now: “well, Mr government procurement officer, looks like ODF is dangerously non-interoperable: here, let me show you how IBM and Sun can’t even agree on basic features; but look, we’ve implemented ISO standard formulas, so we alone avoid that – and you can assess whether we’re doing what we claim – looks like MS Office is the only choice …”)

Personally, I think MS have been disappointingly petty in abandoning the “convention” that the other suites more or less use. I accept that these ODF implementations have limited interoperability and are unsafe for any mission-critical data, but for the benefit of the “Maya’s Wedding Planner” type of scenario, where ODF implementations can actually cut it, I think MS should have included this legacy support as an option, even if they did have to qualify that support with warning dialogs about data loss and interoperability issues.

But - vendors are vendors; it is their very purpose to compete in order to maximise their long-term profits. Users don’t always benefit from this. We really shouldn’t be surprised that we have IBM, Sun and Microsoft in disagreement at this point.

What we should be surprised about is how this interoperability fiasco has been allowed to happen within the context of a standard. To borrow Rick Jelliffe’s colourfully reported words, the whole purpose of shoving an international standard up a vendor’s backside it to get them to behave better in the interests of the users. What has gone wrong here is in the nature of the standard itself. ODF offers an extremely weak promise of interoperability, and the omission of a spreadsheet formula specification in ODF 1.1 is merely one of the more glaring facets of this problem. As XML guru James Clark wrote in 2005:

I really hope I'm missing something, because, frankly, I'm speechless. You cannot be serious. You have virtually zero interoperability for spreadsheet documents.

To put this spec out as is would be a bit like putting out the XSLT spec without doing the XPath spec. How useful would that be?

It is essential that in all contexts that allow expressions the spec precisely define the syntax and semantics of the allowed expressions.

These words were prophetic, for now we do indeed face a present zero interoperability reality.

The good news is that work is underway to fix this problem: ODF 1.2 promises, when it eventually appears, to specify formulas using the new OpenFormula specification. When that is published vendors will cease to have an excuse to create non-interoperable implementations, at least in this area.

Is SP2 conformant?

Whether Microsoft’s approach to ODF was the wisest is something over which people may disagree in good faith. Whether their approach conforms to ODF should be a neutral fact we can determine with certainty.

In a follow-up posting to his initial blast, Rob Weir sets out to show that Microsoft’s approach is non-conformant, subsequent to his previous statement that “SP2's implementation of ODF spreadsheets does not, in fact, conform to the requirements of the ODF standard”. After quoting a few selected extracts from the standard, a list is presented showing how various implementations represent a formula:

  • Symphony 1.3: =[.E12]+[.C13]-[.D13]
  • Microsoft/CleverAge 3.0: =[.E12]+[.C13]-[.D13]
  • KSpread 1.6.3: =[.E12]+[.C13]-[.D13]
  • Google Spreadsheets: =[.E12]+[.C13]-[.D13]
  • OpenOffice 3.01: =[.E12]+[.C13]-[.D13]
  • Sun Plugin 3.0: [.E12]+[.C13]-[.D13]
  • Excel 2007 SP2: =E12+C13-D13

Rob writes, “I'll leave it as an exercise to the reader to determine which one of these seven is wrong and does not conform to the ODF 1.1 standard.”

Again, this is clearly aimed at the slow witted. One can imagine even the most hesitant pupil raising their hand, “please Mr Weir, is it Excel 2007 SP2?” Rob however, is too smart to avoid answering the question himself, and anybody who knows anything of ODF will know that, in fact, this is a tricky question.

Accordingly, Dennis Hamilton (ODF TC member and secretary of the ODF Interoperability and Conformance TC) soon chipped in among the blog comments to point out that ODF’s description of formulas is governed by the word “Typically”, rendering it arguably just a guideline. And, as I pointed out in my last post, it is certainly possible to read ODF as a whole as nothing more than a guideline.

(I am glad to be able to report that the word “typically” has been stripped from the draft of ODF 1.2, indicating its existence was problematic.)

Curious readers might like to look for themselves at the (normative) schema for further guidance. Here, we find the formal schema definition for formulas, with a telling comment:

<define name="formula">
  <!-- A formula should start with a namespace prefix, -->
  <!-- but has no restrictions-->
  <data type="string"/>
</define>

Which is yet another confirmation that there are no certain rules about formulas in ODF.

So I believe Rob’s statement that “SP2's implementation of ODF spreadsheets does not, in fact, conform to the requirements of the ODF standard” is mistaken on this point. This might be his personal interpretation of the standard, but it is based on an ingenious reading (argued around the meaning of comma placement, and privileging certain statements over other), and should certainly give no grounds for complacency about the sufficiency of the ODF specification.

As an ODF supporter I am keen to see defects, such as conformance loopholes, fixed in the next published ODF standard. I urge all other true supporters to read the drafts and give feedback to make ODF better for the benefit of everyone, next time around.

Comments

5/10/2009 5:11:06 PM #

Rob Weir

Alex, you are trying to pull a fast one on your readers.  Shame.  You know very well, of all people, that in an schema that uses XML Schema Datatypes, the word "restriction" has a very specific technical meaning that is not at all like the general English meaning you have lead your innocent readers to believe.

See:  www.w3.org/TR/xmlschema-1/#key-typeRestriction  

Conformance to the ODF 1.1 standard requires both validity with respect to the schema as well as other requirements specified in the text of the standard.  The text explicitly defines additional constraints on cell address syntax which must be met by conformant documents.  It does not appear that Excel 2007 SP2 adheres to these additional constraints.

Rob Weir United States |

5/10/2009 5:29:15 PM #

Jesper Lund Stocholm

Rob,

It does not appear that Excel 2007 SP2 adheres to these additional constraints.

Why do you use the word "appear" here?

Does or doesn't Excel 2007 SP2 - based on your tests - adhere to these additional constraints?

Jesper Lund Stocholm Denmark |

5/10/2009 5:53:16 PM #

Rob Weir

Jesper, I think I've made clear my personal opinion in this on my blog.  But what do you think?  

Rob Weir United States |

5/10/2009 6:49:19 PM #

AndréR

It surprises me that a productive tool is used for these games.

One question is whether Microsoft did “The Right Thing” by users in choosing to implement formulas the way they did. This is certainly a fair question and one over which we can expect there to be some argument.

Whoever did the "Thing" wanted it to be an argument and feels very confident. I wonder if that was a good service to the customers.

Now the prankster begs for spanking. Act your age...    

AndréR Germany |

5/10/2009 7:04:39 PM #

orcmid

It just struck me that the proper way to compare the formulas in Rob's table was with the complete attribute value and (as Rob is quick to argue) not what the user might see in the formula field of the spreadsheet software.

For Excel 2007 SP2, it would be "msoxl:=E12+C13-D13", and similarly for whatever the full one is for all of the other listed implementations.  This is an extension of the great namespace-disambiguation principle to the values of attributes where different implementations (syntax and semantics, it says) are permitted.

Now, there is the suggestion that a great step toward commity would be to recognize the typical form (syntax, maybe not so much semantics) and attempt to let it through, with a little rewriting on the way in and on the way back out.  Perhaps so.  But to assist that to fail to do so somehow constitutes non-compliance with the OASIS ODF 1.1 specification is a great stretch.

I find it useful that Doug Mahugh reminds us of the five guiding principles that were discussed at the first Document Interoperability Initiative workshop where Microsoft presented its intentions for ODF in Office 2007 SP2.  As harsh as the "be predictable" part is in this case, it is difficult to argue that it is not principled and maybe it will be seen to provide a safer path in the long run.  Meanwhile, it would be great to see the OpenFormula draft advanced to public review and adoption as at least a Committee Specification.  That would provide something deemed stable enough for people to start implementing and also developing whatever extra-standard interoperability practices are important.  

The way that bugs and defects in progressive versions of the OpenDocument specification will also require some serious commitment as there become legacy documents with formulas having deviant implementations by one processor or another, and with respect to one edition of the specification or another.

orcmid United States |

5/10/2009 7:09:04 PM #

orcmid

Hmm, ok, now I write assist for insist.  Interesting.  In the past two days I have also mangled hear-here and just received a commercial e-mail that gets form-from wrong.  Sigh.

I see I am also sentences without verbs.  "The way that bugs and defects are resolved in progressive versions ... " is a little better.

But the common trait of all this is that somehow I never see these things until after I press the "Save" or "Publish" or "Send" or "Print" buttons.

orcmid United States |

5/10/2009 7:31:35 PM #

Alan Bell

@Orcmid, I think Microsoft are certainly living up to the "be predictable" principle! In all fairness I don't think anyone expects a version 1.0 product from Microsoft to be fully working.
I do see the logic of namespace disambiguation for formulas where the specification falls short, however stripping out other people's formulas is not very well behaved.
I am of the opinion that a really well behaved application could open Maya's wedding sheet, allow the user to change cell A1 to "Mary's Wedding Planner" and save it without understanding or breaking the formulas in the other cells. I have no idea whether or not any of the ODF applications can or do work that way but clearly Microsoft's one doesn't.

Alan Bell United Kingdom |

5/10/2009 7:44:54 PM #

Alex

@Rob

I'm guessing you're still pretty pumped up after all the gnat swatting you've been engaged in over the last few days but *really* - it's a bit much to imply I'd intentionally mislead my beloved blog readers.

I never took "restriction" to have been used in the XSD sense you invoke. Thinking about it, that is probably because:

1. That is not how the word "restriction" is used in the narrative content of ODF itself, where things are written like: "There may be also restrictions for the values of text formatting properties."

2. The W3C schema document you cite is not referenced by ODF 1.1.

3. ODF never uses the XSD type restriction feature anyway (just the pre-defined types of Part 2 mirrored by some RELAX NG implementations)

4. If this was - as you suggest - merely a gloss on the fact the the "string" datatype is not further restricted, then we'd expect to see that gloss used elsewhere in the schema. We don't; there is a special comment made only for the formula definition.

But gosh, maybe you're right! Who knows? In ODF that is *exactly* the problem.

You are just plain wrong to assert that "the text explicitly defines additional constraints on cell address syntax", as your standards colleagues have already  pointed out to you, there is nothing "explicit" about it once the contexts are taken properly into account. Again, *you* have an interpretation; others have a different interpretation. And again, this is exactly the problem: ODF itself is unclear.

Don't get me wrong: I'm not suggesting you are not entitled to your opinion. But it surprises me that you use your personal interpretation as the basis for judging, and condemning, an implementation of ODF. And it worries me that you deny that any interpretation other than your own can be reasonable, when the ODF spec itself is in such obvious confusion on this topic.

Alex United Kingdom |

5/10/2009 7:46:56 PM #

Alex

@AndréR

You are more than usually elliptical. Who should be spanking whom?

Alex United Kingdom |

5/10/2009 8:08:38 PM #

orcmid

That's a good point, Alan.  I suspect there are no such well-behaving applications, and it is valuable to discuss them and strive for them.

There are tricky conditions around preserving formulas (and other elements and attributes) that are not supported by an implementation.  I suppose an implementation could say that it will not recalculate because there are formulas it doesn't recognize.  But if an user starts editing or adding formulas, I think the only thing to do is ditch the unrecognized ones.  It might warn the user and ask for confirmation before doing that.  

Here, we are making a lot of assumptions about what a processor remembers about the original form of document that was opened, and how it deals with preserving or not preserving material when it passes through the processor and out the other side into the same or even a different format than the original.  I can see a document processor being designed with this sort of thing in mind, but it is difficult to imagine how an existing processor with a single internal model might be able to do this very easily.  (See my initial thoughts about that: orcmid.com/.../...df-interoperability-workshop.asp

Speculating about Excel 2007 itself, I think the problem is that the process of opening a document involves translation into the internal storage structures of Excel, and I doubt they have an already-available way to carry unrecognized material safely through that and out the other side, particularly for formula material.

Still, these are all useful and interesting things to be thinking about and discussing, because this problem won't end soon, not even with the availability of a stable, agreed OpenFormula specification.  I see OpenFormula as opening up the prospect for convergence, but versioning within OpenFormula and handling extensions/corrections of it will also come up at some point.  I would hope that better-behaved processors can also evolve.

orcmid United States |

5/10/2009 8:11:08 PM #

orcmid

OK, this is not one my wrong-wordisms.  I will now give the correct URL in the previous comment, with no punctuation of any kind after it: orcmid.com/.../...df-interoperability-workshop.asp

orcmid United States |

5/10/2009 8:22:20 PM #

Doug Mahugh

Alex, I understand your point that we could have added an option to support undocumented and already non-interoperable (between Symphony and OO) formula syntax as a fallback option, with warning dialogs about data loss and interoperability issues.  However, given the climate we live in – where the co-chair of the ODF TC spends much of his time publicly attacking our products and questioning our motives – I’m having a hard time believing that he wouldn’t merely shift his attack to demonstrations of how data can be lost with SP2, using the exact same sort of examples that I’ve provided for Sympony and OpenOffice.org.

In any event, it seems very telling that there has been a round of strong reactions to my post from Rob and his friends, without one single world about the core ODF formula interoperability problems that existed before SP2 was around, and continue to exist today.  In the current debate, it seems Microsoft is the only entity arguing that ANY risk of formula data loss is unacceptable within the context of standards-based interoperability.  We have many customers who agree with that sentiment, and I’d imagine other vendors have  customers who feel the same way.

Doug Mahugh |

5/10/2009 8:48:59 PM #

Alex

@Doug

Well, even if MS discovered the cure for cancer there'd still be people finding something to blame you for about for it...

- Alex.

Alex United Kingdom |

5/10/2009 9:50:45 PM #

carlos

Alex, i know that you usually want to be "neutral", i.e: if you are critic with Microsoft, then you give another one to the open source community or "IBM". But here, we have clear different "attitudes" regarding practical interoperability:

Doug Mahugh: "This is the state of formula interoperability among ODF spreadsheets today" [0], translated: "there are many problems with ODF interoperability so, we don't give a s... and will aggravate the problem and won't put the damn square brackets in the formulas".

Robert Weir: "I must admit that I'm disappointed by these [interoperability test results]. This is not a step forward compared to where we were two months ago. [...] I have some suggestions for how to move things forward again" [1]

Clearly distinct approaches to the problem.


[0] blogs.msdn.com/.../1-2-1.aspx

[1] www.robweir.com/.../...on-excel-2007-sp2s-odf.html

carlos United States |

5/10/2009 10:14:15 PM #

Alan Bell

@orcmid, yes, that kind of preservation of unrecognised bits in a file format is a new way of thinking for applications where the application isn't allowed to assume it owns the file format. It isn't really that unheard of, it is what XML is all about really. When something like MusicML gets incorporated into ODF (My guess would be in ODF 1.4) applications not yet implementing musical notation shouldn't trample all over someone's sonata (or symphony for that matter). Having a file format that is deeply tied to an implementation design (some file formats are not much more than a serialisation of the in-memory objects) is quite unhealthy in terms of robustness and good behaviour, as the saying goes applications should be liberal in what they accept and conservative in what they produce.
I am actually not that upset about the spreadsheet situation, for one thing I don't use Microsoft Office and I don't tend to share spreadsheets with those who do, so from my selfish perspective, I don't care. From a more outward looking perspective clearly spreadsheets don't work (valid, conformant or otherwise - the bottom line is that they don't work). To me this just underlines the importance of working together to get ODF 1.2 through the standardisation process so that applications can improve interoperability and functionality as time goes by.

Alan Bell United Kingdom |

5/10/2009 11:36:49 PM #

Doug Mahugh

Carlos, we couldn't arbitrarily change the syntax of IS29500's formula markup language to match the undocumented informal agreement that exists among certain ODF implementers, because if we did then the namespace prefix would no longer specify "the syntax and semantics used within the formula" as required by Section 8.1.3.  You're asking why we didn't deviate from the standard here, and the answer is simply that "Adhere to the ODF 1.1 standard" is our top-priority guiding principle.

Doug Mahugh |

5/11/2009 12:02:24 AM #

carlos

Doug, your words make me remember the words of Brian Jones when people asked "why there is no native support of ODF in Office?". He always answered "there is no customer need of this functionality" ( meanwhile , lot of governments and public organizations were choosing and mandating ODF all around the world! )

I mean: your words sound as a pure political/formal/marketing answer that fails to explain the basic fact:

------------------
Fact: Microsoft is the only one of seven main ODF implementations that fail to achieve interoperability in ODF formulas:


# Symphony 1.3: =[.E12]+[.C13]-[.D13]
# Microsoft/CleverAge 3.0: =[.E12]+[.C13]-[.D13]
# KSpread 1.6.3: =[.E12]+[.C13]-[.D13]
# Google Spreadsheets: =[.E12]+[.C13]-[.D13]
# OpenOffice 3.01: =[.E12]+[.C13]-[.D13]
# Sun Plugin 3.0: [.E12]+[.C13]-[.D13]
# Excel 2007 SP2: =E12+C13-D13
------------------

This is a pure practical fact, there is no need to a long post with screenshots like this [0].

But i know, you have won 1 year of time with this trick ( like the ODF native support thing ). Sooner or later you will change this to properly support the ODF standard and you will announce that with a great press release [1] "Micorosoft enhances customer choice and interoperability now *really* producing interoperable ODF documents"

Conclusion: we live in a generous world. And actually, we customers deserve what we have.

[0] blogs.msdn.com/.../1-2-1.aspx

[1] www.microsoft.com/.../05-21ExpandedFormatsPR.mspx

carlos United States |

5/11/2009 12:42:16 AM #

Rob Weir

Alex, ODF 1.1 references XML Schema Part 2, which uses "restriction" in the same sense that I do.

As for additional constraints, how to you explain away 8.3.1: "In some cases it is not necessary to provide the name of the table. However, the dot must be present."  Or for that matter any part of 8.3.1.  It may not be in the preferred ISO drafting format, but ODF 1.1 is an OASIS Standard and is not obliged to be in ISO format.  The language in 8.3.1 is straightforward and unambiguous English, presumably a language not unfamiliar to you.   So what part of 'must' is tripping you up?

Rob Weir United States |

5/11/2009 2:11:40 AM #

AndréR

@Alex: Exactly!

Who should be spanking whom?

Who: Probably those who feel provoked.
Whom: agent provocateur

AndréR Germany |

5/11/2009 6:25:55 AM #

Jesper Lund Stocholm

Rob,

Jesper, I think I've made clear my personal opinion in this on my blog. But what do you think?

It is clear to me that ODF 1.1 leaves sufficient room for interpretation on how to reference cells in tables. I therefore think you should stop bitching about ODF 1.1 conformance and tend to what really matters in this discussion - interoperability between ODF-implementations and how to best achieve this.

Smile

Jesper Lund Stocholm Denmark |

5/11/2009 7:53:19 AM #

Alex

@Rob

- You think the "must" trumps the "typically";

- Microsoft thinks the "typically" trumps the "must";

- I doubt any semantic constraints in ODF are normative.

The spec should not leave this room for interpretation: get it fixed!

Alex United Kingdom |

5/11/2009 8:31:59 AM #

Rob Weir

Alex, if Microsoft thinks that the scope of "typically" crosses several full stops, and into a different paragraph, and even into a different section and trumps a "must" then their reading is aberrant.  Just because one can misread a text does not make that reading valid.  If a text can be read as consistent than you must take that reading.  Throwing out words like "must" is not something you do.  Similarly throwing out "typically" is not permitted.  You need to take a reading that is consistent, both with the text, with grammar and with the context.

In any case, we're not talking about a semantic constraint here.  We're talking about a syntactical constraint on cell address notation.  So you're avoiding the question.

I'll put it to you another way.  Does your reading permit an implementer to write out a formula that is syntactically malformed XML, say including internal quotation marks that are not not escaped as entities?  Why or why not?

Rob Weir United States |

5/11/2009 8:56:12 AM #

Alex

@Rob

I'm not going to argue with you - I don't particularly favour any reading. It's just I notice different reasonable and qualified people (and not just MS) interpret this differently. That indicates the standard is faulty.

In general of course provisions "cross" full stops, paragraphs, even whole sections. So the conformance provisions in s1.5 apply to an entire XML document (which answers your last question).

The very fact that you are having to weave these arguments in favour of your reading, and argue against others, indicates that there's a problem here. But of course neither my blog nor your blog in a normative annex to ODF, no matter how much you might wish that to be the case.

Alex United Kingdom |

5/11/2009 9:02:49 AM #

AlexH

Honestly, who cares whether or not MS have correctly read the spec. with regards putting square brackets around cell addresses? For all except the most trivial of trivial spreadsheets, it makes no difference - the serious problems around function naming/availability, and making sure stuff works correctly together means that interoperability is essentially holed below the water line anyway. The square brackets are an unamusing side-show.

The work the ODF TC have done on OpenFormula is wonderful: it does what MS should have done in OOXML (and what OASIS should have done in the first place), but goes way beyond and becomes testable.

But ye gods, are we going to have to wait a year before OASIS poops out an approved spec.? Might we see this from ISO in a couple years time?

This is such an important, critical, feature - can't the OASIS TC just pop the OpenFormula spec through the approval process on it's own and get it ratified? It must be complete, because people are using it right now - and if ODF 1.2 is still out in the weeds somewhere, put out ODF 1.1.1 which references the new formula spec. or something (obviously, it can't be 1.2, because vendors are already shipping that - what a mess).

AlexH United Kingdom |

5/11/2009 3:19:27 PM #

carlos


Microsoft: you are the only one in seven implementations that doesn't put the square brackets. Just fix it , if you really care about customers and interoperability.

Here you are the ascii codes including the ones of "[" "]"

http://www.asciitable.com/
http://www.unicodetables.com/

... and no, you won't find this codes *explicitely* in the ODF 1.1 specification, but trust me, this are the one used by the main office implementations.

carlos United States |

5/11/2009 3:33:00 PM #

Doug Mahugh

Rob, it may be helpful to hear the opinions of other members of the ODF TC on your theory that the cell-referencing syntax is normatively specified in ODF 1.1.  I had hoped to bring the issue up on this morning's ODF TC call, but after 30 minutes on hold I've given up, and I just now saw that today's call was cancelled a few hours ago -- my mistake.  Maybe next week.

Doug Mahugh |

5/11/2009 4:56:20 PM #

marc


I tried the formula-thing in Gnumeric creating and saving a simple spreadsheet. The result: Gnumeric saves formulas in the interoperable way ( contrary to what the Microsoft product does ):

"oooc:=[.B3]+[.C3]+[.D3]"

~ $ gnumeric --version
gnumeric version '1.8.2'

marc United States |

5/11/2009 5:20:00 PM #

AlexH

marc: I tried saving a file from OpenOffice.org and opening it in Gnumeric to see this "interoperability" in action.

Result - large warning dialog, and my formula is completely nuked. Great.

AlexH United Kingdom |

5/11/2009 6:37:42 PM #

marc

@AlexH, i created a document in OpenOffice ( version 2.4 , Linux Mint 5 Elyssa ) , saved and opened with gnumeric 1.8.2 and didn't have any problem.

The formula is stored as:

"oooc:=[.B3]+[.C3]+[.D3]"

By the way, gnumeric support of ODF is still in progress ( when you choose "save as" you will see a "OpenDocument__UNFINISHED" file type ), so this confirm my point: even in incomplete implementations the way-to-go is to put the square brackets ! put the square brackets ! put the square brackets ! put the square brackets !



marc United States |

5/11/2009 9:27:45 PM #

pingback

Pingback from blogs.technet.com

Gray Matter : Clearing up a few matters with respect to ODF and SP2

blogs.technet.com |

5/12/2009 4:25:23 AM #

marc

By the way, Zoho sheet ( http://sheet.zoho.com ) stores the formulas as:

"oooc:=[.B2]+[.C2]+[.D2]" ( i tried it with a very simple spreadsheet )

and the same goes for EditGrid.com .

So, the list showing how various implementations represent a formula is:

# Symphony 1.3: =[.E12]+[.C13]-[.D13]
# Microsoft/CleverAge 3.0: =[.E12]+[.C13]-[.D13]
# KSpread 1.6.3: =[.E12]+[.C13]-[.D13]
# Google Spreadsheets: =[.E12]+[.C13]-[.D13]
# OpenOffice 3.01: =[.E12]+[.C13]-[.D13]
# Sun Plugin 3.0: =[.E12]+[.C13]-[.D13]
# Gnumeric 1.8.2: =[.E12]+[.C13]+[.D13]
# Zoho sheet: =[.E12]+[.C13]+[.D13]
# www.editgrid.com: =[.E12]+[.C13]+[.D13]
# Excel 2007 SP2: =E12+C13-D13 (WTF????)

marc United States |

5/12/2009 7:10:45 AM #

JCGA

@Dough Mahugh

"Rob, it may be helpful to hear the opinions of other members of the ODF TC on your theory that the cell-referencing syntax is normatively specified in ODF 1.1."

Doug, read the ODF v1.1 spec please.
Particularly, read section: 8.3.1 Referencing Table Cells
That section is not marked as non-normative.

JCGA Mexico |

5/12/2009 7:15:56 AM #

hAl

@marc
Why don't you list the namespaces to that list of implementations.

As only similar namespaces would suggest interoperability between the formala languages.

It is as simple a van be. With OOo 2.x being the main ODF 1.1 implementation all other have follewed and reused the OOo propriety formula's using the OOo namespace.
MS has chosen to use the Excel formula's and chose to use the ISO/IEC standardized formula's from Office Open XML for that.

So for now I see a lot of people presenting a list of ODF implementation with propriety OpenOffice.org/StarOffice formula's extensions as more interoperable then and ODF implementation using an ISO/IEC standardized formula language.
That is just funny.

It is like people supporting the current proprety .doc/.xls/.ppt because most implementation support those in an interoperable way. (a lot more interoperable than it seems current ODF implementations are).

This whole discussion on 5 or 6 blogs is only supporting that people/ organizations/governments should keep using the current binary files for the next 5 years and should stay away from standards. (or should I say "must" in stead of "should").

hAl |

5/12/2009 8:04:09 AM #

Alan Bell

I wonder what would happen if I wrote an implementation with a formula language based on LOLCODE, it might look something like this:
"lolxl:=CAN I HAZ [.E12] UP [.C13] NERF [.D13]"
which would I think scrape through the ODF 1.1 requirements, but would, I suspect, be an interoperability challenge to all the implementations. I think they should be required to preserve my lolcode formula for a simple load and save round trip.
Personally I can totally see why Microsoft didn't use the OpenOffice.org namespace and old formula language. I don't see why they couldn't have used the OpenFormula draft, but as they had the OOXML syntax kicking about already they shoved that in.
I think on loading a spreadsheet an application should alert the user (or provide means for the user to fine out) about formulas it cannot evaluate and also alert the user to cells that evaluate to a different result than the one they were saved with (which is going to be OK sometimes and not OK other times)

Alan Bell United Kingdom |

5/12/2009 1:38:30 PM #

Jesper Lund Stocholm

Alan,

I don't see why they couldn't have used the OpenFormula draft

Because that is just what it is - a draft. With SP release-cycles for e.g. Microsoft Office being as long as they are, I am sure none of us want formulas based on drafts - drafts that are not even public drafts at the moment.

Drafts are fine for testing purposes, but having OOo as default setting produce ODF 1.2 markup that is also just a draft it fundamentally stupid - and I am surprised that Rob has not cast his wrath upon Sun for this yet.

Jesper Lund Stocholm Denmark |

5/12/2009 2:02:42 PM #

Alex

@Jesper

Rob did cast his wrath (well, raised an eyebrow) at OO.o for this, when he wrote:

Sun should write out formulas in ODF 1.1 format, using the legacy "oooc" namespace prefix that the other vendors are using. Remember, the other vendors are using that namespace specifically for compatibility with OO's ODF documents. This is the current convention. To unilaterally switch, without notice or coordination, to a new namespace, is not cool. When ODF 1.2 is an approved standard, then we all can move there in a coordinated fashion, to cause users minimal inconvenience.” ( http://is.gd/wxLT )

While Sun is described as "not cool", MS is however guided by "malice or incompetence". It's an interesting reflection on the clue-level (or prejudice) of the hatebois that they haven't picked-up on Sun's interop breakage here ...

Alex United Kingdom |

5/12/2009 2:21:01 PM #

Jesper Lund Stocholm

Alex,

While Sun is described as "not cool", MS is however guided by "malice or incompetence". It's an interest commentrary on the clue-level (or prejudice) of the hatebois that they haven't picked-up on Sun's interop breakage here ...

I noticed that - but it seemed to me that it was just put in "for good measures" as I think someone put it earlier this week.

If Sun wants to play around with ODF 1.2 they are more than welcome - that is what drafts are for. But they should make it an opt-in choice for the user (and keep defaulting to ODF 1.1) and not use it as the default file/formula format.

I agree with Rob that using spreadsheet formulas in ODF defined in OOXML hurts interoperability - but there is no way you could argue that Suns usage of ODF 1.2 hurts it any less. That Rob barely mentions it (and does not dedicate a few full articles about it slamming their decisions) really tells you a lot.

I have no problems with Rob serving as ODF TC co-Chair - I think he does a great job at it. But I do think that his actions (or lack thereof) underlines the sentiment that some has: that not only is Rob not vendor neutral - he is primarily anti-Microsoft.

(and I am aware that some think that it is a fantastic quality)

Smile

PS: "Hatebois" - another one of your fancy British words that I (and wiktionary.org) don't understand. (Hate-boys?)

Jesper Lund Stocholm Denmark |

5/12/2009 2:43:13 PM #

Alex

@Jesper, "fanboy" is often written "fanboi"; I think that gives it more of an edge (see http://is.gd/zaaQ for some usage). So yes.

Alex United Kingdom |

5/12/2009 3:36:15 PM #

Alan Bell

@jesper, I can see reasons for and against using draft openforumula. I can see reasons for and against trying to be compatible with OpenOffice legacy formulas. I can see reasons for and against LOLCODE. Believe it or not I can see reasons for and against putting in whatever they did put in under a new namespace. None of these reasons are definitive. I take your point about OpenFormula being a draft, however I see it in practical terms and I would say that implementing a spec that is 100-ε% complete is pretty similar to a 100-ε% implementation of a spec that is fully complete. For small values of ε.
ODF 1.1. without a formula specification isn't feature complete for spreadsheets. This episode just brings out the need for 1.2 to be released.

Alan Bell United Kingdom |

5/12/2009 3:50:03 PM #

Alex

@Jesper

You wrote:

"I have no problems with Rob serving as ODF TC co-Chair - I think he does a great job at it. But I do think that his actions (or lack thereof) underlines the sentiment that some has: that not only is Rob not vendor neutral - he is primarily anti-Microsoft."

Agreed on Rob's chairmanship. Whatever disagreements I might have with Rob, and whatever Rob's stance towards his employer's competitors, I do think it is quite wrong for Microsoft to comment on his suitability to chair the ODF TC. Throughout this whole Office standards saga there have been plenty of rumblings from more than one US corporation about who is suitable for what position, and this kind of corporate meddling should, I believe, be strongly resisted.

There are processes within standards organisations to cope with complaints of this kind. If need be, let those be followed.

Alex United Kingdom |

5/12/2009 3:53:39 PM #

Jesper Lund Stocholm

Alan,

ODF 1.1. without a formula specification isn't feature complete for spreadsheets. This episode just brings out the need for 1.2 to be released.

I totally agree Smile

Jesper Lund Stocholm Denmark |

5/12/2009 3:56:53 PM #

Alex

@Alan @Jesper

> I totally agree

+1

Consensus breaks out Smile

Alex United Kingdom |

5/12/2009 7:10:50 PM #

marc

Jesper said:

"That Rob barely mentions it (and does not dedicate a few full articles about it slamming their decisions) really tells you a lot."

Rob Weir said [0]:

"Sun should write out formulas in ODF 1.1 format, using the legacy "oooc" namespace prefix that the other vendors are using."

"I know OO 3.01 has an option to save in ODF 1.0/1.1 format. IMHO, this setting should be the default. I'm not sure if the Sun Plugin has a similar configuration option, but I hope it does."

I don't think this is a "barely mention". This is a clear indication that things need to be improved and Rob shows how.

Microsoft , on the other way, responded with a direct attack to Rob Weir work in ODF.

I would like to hear Microsoft responding:

"Yes, there is a problem here, we are the only one of eight implementations that don't put square brackets in formulas and we generate a lot of red cells in Weir's table, so we will add an option in Office 2007 to save formulas a la "openoffice way" to keep interoperability , until OpenFormula is standardized."

Instead, Microsoft responded [1] something like this:

"ODF interoperability is a mess ( why? well... because ... because yes! see: if you format a number as text and then put it in a formula you have different results in two implementations ) , so we will not improve it and we will output ODF formulas in a totally no interoperable way, because if you format a cell number as text "


[0] www.robweir.com/.../update-on-odf-spreadsheet.html
[1] blogs.msdn.com/.../1-2-1.aspx

marc United States |

5/12/2009 7:48:28 PM #

Jesper Lund Stocholm

marc,

Rob Weir said [0]:

"Sun should write out formulas in ODF 1.1 format, using the legacy "oooc" namespace prefix that the other vendors are using."

"I know OO 3.01 has an option to save in ODF 1.0/1.1 format. IMHO, this setting should be the default. I'm not sure if the Sun Plugin has a similar configuration option, but I hope it does."

I don't think this is a "barely mention". This is a clear indication that things need to be improved and Rob shows how.


Yeah, he is sooo putting his foot down Smile

Jesper Lund Stocholm Denmark |

5/13/2009 6:51:12 AM #

Paul E. Merrell, J.D. (Marbux)

@Jesper ""I have no problems with Rob serving as ODF TC co-Chair - I think he does a great job at it."

Personally, I think all chairmen of the ODF TC and Ecma TC 45 should step down and be replaced by a single probation officer. Smile But I guess SC 34 is as close to that as we're likely to see any time soon.



Paul E. Merrell, J.D. (Marbux) United States |

5/13/2009 8:36:53 AM #

Jesper Lund Stocholm

Hi Paul,

Amazing ... after two years now I (finally) have a face to match the name.

Smile

Jesper Lund Stocholm Denmark |

5/13/2009 3:26:12 PM #

marc

wow ! This is the famous "man without a garage" !! Wink

marc United States |

5/13/2009 6:18:39 PM #

AndréR

Dear Marbux,

I read your Universal Interoperability Council comments on EIF v2. I note that you felt invited to comment on European policies.

As of the definition of the term "interoperability", here you have to understand the specific context and mandate of the program, the legal base: paneuropean cooperation of national administrations. Quite a few weeks ago the followup program for IDABC was set up and here again the term "interoperability" is hardcoded in the program.

Article 2: ""interoperability" means the ability of disparate and diverse organisations to interact towards mutually beneficial and agreed common goals, involving the sharing of information and knowledge between the organizations,  via the business processes they support, by means of the exchange of data between their respective ICT systems;"

Fortunately the attempt to hardcode the term "Open Standards" failed. It would have been a bit "over the top" risky, so I was glad that it was shot down.

Please also note that Council Directive 91/250/EEC was recently consolidated (codification), and its objective is copyright protection of software.

I would prefer as you do a more "technical" interoperability term. However, the main challenges are in the field of "legal interoperability". The tendency of broadness is to turn "interoperability" into a term as "cooperation" and sell the whole SaaS consultancy vapourware and portals.

//André

Unfortunately the low key proposal for the followup program to focus on "promotion of open document exchange format" was also not accepted but I see from your submission that you aim in that direction.


AndréR Germany |

5/14/2009 6:34:27 AM #

Paul E. Merrell, J.D. (Marbux)

@Jesper and Marc: Yes, I finally slowed down long enough to RTFM on how to load a gravatar.  

@André: Unfortunately, those comments were a fairly hasty job. But the suggestion that the EIF's interoperability definition needed tweaking to better address human < > machine interoperability, particularly in the area of accessibility, was intended to improve the emerging definition of an "interoperability framework." Much of that is flowing from a United Nations program designed to assist nations and drive toward some uniformity as governments attempt to clean up the legacy mess left by decades of uncoordinated government IT infrastructure development. The EIF has the honor of being the government interoperability framework whose documentation is most mature,  

Council Directive 91/250/EEC has a legal significance beyond copyright. It provided the crucial definition of "interoperability" in the Court of First Instance's ruling in Commission v. Microsoft, a definition that practically mirrors verbatim the JTC 1 definition. So I thought it important that the EIF not undermine those definitions with a highly ambiguous definition carried over from EIF 1. The need for harmonization often crops up in unexpected places. So I suggested a separate definition for "technical interoperability," a term used but left undefined in the draft.

I can happily report that I studied the revised version of the Directive right after it was released and its interoperability provisions have obviously been updated in light of the Court of First Instance ruling. No fundamental change was wrought on those provisions, but the added detail is, I think, welcome. (I'm aware that other provisions of the revised Directive have come under serious and seemingly deserved criticism.)  

On the term "open standard," I think it's outlived all utility. All of the recent definitions I've looked are full of loopholes and clash with the governing international law. I have a nascent effort to define a substitute term and definition at www.universal-interop-council.org/specification  It attempts to encapsulate, restate, and reference  the law governing interoperability in standards work, with a bias toward universality in the areas where the law allows discretion.

I caution, however, that it's still weak in regard to accessibility issues. In the next draft, I'm aiming to work in the requirements of the U.N. Convention on the Rights of Persons with Disabilities, which took effect around most of the globe last May. http://www.un.org/disabilities/index.asp I see considerable advantage in getting the human < > machine interface recognized as a dimension of  "interoperability framework" rather than its present after-thought treatment given in the draft EIF 2.  

I didn't expect my suggestions to be embraced in the next draft. Bureaucracies don't work that quickly once they've arrived at a draft. But I thought it important to get those concepts into the mix to simmer for a later revision.

And yes, I'm after open document exchange formats. IBM claims to have invented the word processor in 1956. One can question the originality of the invention, but in my view 53 years is more than enough time for IBM to restore the open standards-based word processing interoperability Big Blue trashed when it embraced and extended the Teletypesetter telegraphy code to first successfully commercialize computerized word processing in the North American newspaper industry. I was a newspaper typographer in my first career and was there when the extend part happened. My patience with Big Blue's seemingly endless excuses for further delays in fixing the ODF standard is very thin. Smile


Paul E. Merrell, J.D. (Marbux) United States |

5/14/2009 10:13:54 AM #

Paul E. Merrell, J.D. (Marbux)

@Rob Weir: ""I know OO 3.01 has an option to save in ODF 1.0/1.1 format. IMHO, this setting should be the default."

A bit misleading there, unintentional no doubt. True, OOo 3.01 has an option to set what is labeled as "ODF 1.0/1.1" as the default file save type. But you might as well scrub ODF 1.0 from the picture. It's a setting for a single format and it is ODF 1.1, not ODF 1.0." The only time the app can write to ODF 1.0 is when features unique to ODF 1.1 are not written.

As for setting it as the default as Rob recommends, the same options setting dialog advertises: "Not using ODF 1.2 may cause information to be lost." Sun should just set this format as the default? Sounds like under-specified advice from Rob to me.

So like Microsoft Office, OOo 3.01 simply gives you a warning rather than setting a compatibility mode that will limit the app's feature set when composing an ODF 1.1 document. I think it hardly better that Microsoft flashes the warning at you when saving the file and OOo does not. Warnings are simply not an adequate solution.

My favorite feature of the W3C Compound Document by Reference Framework is this compatibility requirement that forbids that kind of lossy format support:

"A conformant user agent of a superset profile specification must process subset profile content as if it were the superset profile content."

www.w3.org/TR/2007/CR-CDR-20070718/#conformance

Of course ODF has never been profiled, so there is work to be done before a similar conformance clause could be added to ODF.

But I don't think a mere "may lose data" warning comes even close to fulfilling the market and legal requirement for interoperability. This is an example of broken human < > machine as well as machine < > machine interoperability. It's also a defect in the ODF specification drastically in need of repair. The ODF 1.1 conformance section still bluntly reads in relevant part:

"There are no rules regarding the elements and attributes that actually have to be supported by conforming applications ..."

Coupled with the specification's in blanc check to willy nilly create foreign elements and attributes and countless other gaps in the specification, it is truly a challenge to argue persuasively that ODF is anything other than a de facto standard disguised in the clothing of a de jure standard.

The same is true of OOXML but one might hope that we are nearing the end of the era when all those who report bugs in the ODF specification are painted as garage-deprived or as Microsoft agents. ODF is as drastically in need of bug reports and fixes as OOXML.

Paul E. Merrell, J.D. (Marbux) United States |

5/14/2009 2:06:03 PM #

AndréR

"...one might hope that we are nearing the end of the era when all those who report bugs in the ODF specification are painted as garage-deprived or as Microsoft agents. ODF is as drastically in need of bug reports and fixes as OOXML."

Fully agree with regard to ODF 1.2. The progress on Openformula is not very encouraging but looks more like standstill:
www.oasis-open.org/.../documents.php

I am sure the review process can be crowdsourced but you need a somewhat mature draft to start with, or at least attached capacity to editorial work.

For smaller texts we have now a very nice web 2.0 comment tool http://www.co-ment.net/

AndréR Germany |

5/18/2009 4:45:30 AM #

ghomem

@Alex,

I read this post carefully and your desire to remain neutral seems quite forced.  The reason your neutrality sounds forced is: good faith is *not* a relative concept. On this particular discussion you can't just be cool with both sides.


The fact is: no matter how good a standard is, there will always be windows open for abuse. Those will be abused by whichever implementer desires so.

If Microsoft had followed the same convention others do (despite the existing problems with the standard) I don't see how they could be criticized. They'd be praised if, for the first time, their actions matched their public interoperability announcements.

@Doug,


"However, given the climate we live in – where the co-chair of the ODF TC spends much of his time publicly attacking our products and questioning our motives – I’m having a hard time believing that he wouldn’t merely shift his attack to demonstrations of how data can be lost with SP2, using the exact same sort of examples that I’ve provided for Sympony and OpenOffice.org."


I don't know what climate you're seeing on the ODF TC but this statement suggests decisions are taken emotionally and not based on "common sense". Common sense suggest supporting "Maya’s Wedding Planner" and "Joe's Beer Planner".

Even in the worst case scenario you refer to... I don't see how "demonstrations of how data can be lost" on specific intricate scenarios could be worse than "demonstrations of how basic interoperability doesn't work".

ghomem |

Comments are closed

About the author

Alex Brown


Links

Legal

The author's views contained in this weblog are his, and not necessarily of any organisation. Third-party contributions are the responsibility of the contributor.

This weblog’s written content is governed by a Creative Commons Licence.

Creative Commons License     


Bling

Use OpenDNS  

profile for alexbrn at Stack Overflow, Q&A for professional and enthusiast programmers

Quotable

Note that everyone directly involved in the development of ISO standards is a volunteer or funded by outside sponsors. The editors, technical experts, etc., get none of this money. Of course, we must also consider the considerable expense of maintaining offices and executive staff in Geneva. Individual National Bodies are also permitted to sell ISO standards and this money is used to fund their own national standards activities, e.g., pay for offices and executive staff in their capital. But none of this money seems to flow down to the people who makes the standards.

Rob Weir

RecentComments

Comment RSS