a r t i c l e s   >    s c o p e   c o n t r o l l e d   d a t a

summary of anger

Ive always said, I dont hate people who are stupid... just stupid people who think they're smart.

That's why I hate XML.

Not because it has no value - simply because so many people make claims that it solves more problems than it actually does. That's why one day I decided to think about its limitations, and implement something that does at least what XML does, but achieves more.

XML is failing to be the 'silver bullet' way to format arbitrary data for three main reasons:

  1. XML does not handle table based data very well, which is the primary data format for business applications
  2. XML does not handle unformatted data well, such as binary files, or any kind of serialized data, of which there is a great deal in the world (images, zip files, movies, sounds)
  3. To use XML, everything must be turned into an object. This adds bloat as well as confusion to things that are not objects - such as relational database tables, and files.

XML does have its advantages, in that it is a flexible enough language for display of many kinds of content, but for data storage it simply does not work well. What is needed is a format that can give us the flexibility and readability of XML, with a smaller footprint, and yet still allows us to send table- and tree-structured data, and completely unstructured data.

The goals for a replacement for XML, no matter what it is, should follow many of the same goals of the XML project. However, it must be better than XML in every way. The goals I see are as follows:

  • It must be human readable
  • It must be extensible
  • It must be internationalizable
  • It must be parseable by existing applications
  • It must support easy transformation into other formats, primarilly XML
  • It must natively support binary data
  • It must natively support common data structures

enter the SCAT

After using XML and SOAP extensively , I started thinking about how I would have done it differently. After a bit of pondering, I came up with a very preliminary draft of an alternative. I called it SCAT, short for Scope Controlled dATa.

Below is a small sample of what I think could be a viable replacement for XML. As far as I can tell, it seems to do a good job of all of the above, while remaining terse, and readable. As an added bonus, it might even be easily parsed with tools such as grep, sed, and awk.

Im still working on the specifics of SCAT. My goal is to solve this problem backwards by having a loose spec first, make tools for it, test them, see where it breaks, then fix the spec.

Please remember that this format is very new... the main reason I did it was so that I could complain about XML more loudly ;)


example

A SCAT file must be able to tersely describe relationships, and name-value pairs. The majority of XML data I see falls into this kind of category. It must also be able to support raw binary data, and specific data types. A sample SCAT file would look like this:

SCAT.encoding=iso-8859-15
SCAT.encoding.magic=??????
company.division.customer
.id=99999
.fname=Joe
.lname=Blow
.company=joeblow.com
.request.text.SCAT.TYPE=TEXT
-----12345
Im curious about an error message I got while using your
product. Im sending a screenshot along for you to analyze.
-----12345
.image.SCAT.ID=9999
.image.SCAT.TYPE=DUMP
.image.content-type=image/gif
.image.content-length=272
-----12345
gif9983794987162987346918723698713948763298471632874617264987
1827309841720384709182374082174987230984729834701982374098174
1234871837401823709872138710983274832740973240982740982709756
5123085753245392170984387509873508370274380923874098237075487
1273897409817203874018273498
-----12345

This SCAT file is a collection of SCAT objects, most of which have a type of 'STRING'. There is one main object in this request, a company.division.customer object. All other lines reference sub-objects to this object, and their string, TEXT, or binary DUMP values.

This is better than XML for a SOAP-like protocol for several reasons:

  • its more terse
  • its more human readable
  • its machine parsable with existing Unix tools
  • all objects are streamable
  • it makes no distinction between sub nodes and attributes
  • internationalization is handled by both an encoding name, and a magic string that can only be parsed if that encoding is true.

read the spec in progress...