Annoyance-Driven Blogging

This post is, in fact, not quite like the others. It is a parody of Hypercritical I wrote for parody week, so take it with a big grain of salt…

I’ve been reading Hypercritical, John Siracusa’s new blog outside of Ars Technica, and it has been good to read more of John, rather than the glacial pace his blog there had been updating lately.

But even on his own space, John has been unable to escape some of the trappings of his past. A blog that updates with some frequency naturally lends itself to multi-post reading sessions. But reading a post about the annoyance of having to watch a minute and a half of opening credits before each episode can get tiresome.

To be fair to John, the existence of this kind of post may not be entirely under his control, given his quasi-OCD tendencies. But getting bogged down in these details misses the point.

Yes, we all know and love John Siracusa for his, well, hypercritical tendencies, but these are best consumed as part of a post on a broader subject, like a spice, having nothing but that in a post quickly gets to be too much.

This may sound comically selfish, but true innovation comes from embracing your audience expectations, not fighting them. Find out what is annoying your readers. Give people what they want and they will beat a path to your door.

We nerds love bickering about technology for its own sake. Indeed, there’s always something to be gained by criticizing the state of the art and goading into providing more of a good thing. But the most profound leaps are often the result of applying criticism as strictly needed in the context of a more constructive post. By all means, criticize, but also research, expose and propose what could be done better and how. Go after those things and you’ll really make people love you. Accentuate the positive. Eliminate the negative.

How does ETC work? A sad story of application compatibility

This post is, in fact, not quite like the others. It is a parody of the Old New Thing I wrote for parody week, so take it with a big grain of salt…

Commenter Contoso asked: “What’s the deal with ETC? Why is it so complicated?“

First, I will note that ETC (which stands for Coordinated Eternal Time) is in fact an international standard, having been adopted by ISO as well as national and industrial standard bodies. The specification is also documented on MSDN, but that’s more for historical reasons than anything else at this point, really. But okay, let’s discuss ETC, seeing as that’s what you want me to do.

ETC is not complicated at all if you follow from the problem to its logical conclusion. The youngest among you might not realize it, but the year 2000 bug was a Big Deal. When it began to be discussed in the public sphere starting in 1996 or so, most people laughed it off, but we knew, and always knew that if nothing was done computers, and all civilization in fact, would be headed for a disaster of biblical proportions. Real wrath of God type stuff. The dead rising from the grave! Human sacrifice! Dogs and cats living together… mass hysteria!

The problem originated years before that when some bright software developers could not be bothered to keep track of the whole year, and instead only kept track of the last two digits; so for instance, 1996 would be stored as just 96 in memory, and when reading it it was implicitly considered to have had the “19” before it, and so would be restored as “1996” for display, processing, etc. But it just happened to work because the years they saw started in “19”, and things would go wrong as soon as years would no longer do so, starting with 2000.

What happened (or rather, would have happened if we let it happen, this was run in controlled experiment conditions in our labs) in this case was that, for starters, these programs would print the year in the date as “19100”. You might think that would not be too bad, even though that would have some regulatory and other consequences, and would result in customers blaming us, and not the faulty program.

But that would in fact be if they even got as far as printing the date.

Most of them just fell over and died from some “impossible” situation long before that: some would take the date given by the API, convert it to text, blindly take the last two digits without checking the first two, and when comparing with the date in its records to see how old the last save was would end up with a negative age since it did 0 – 99 as far as the year was concerned, and the program would crash on a logic error; others would try and behave better by computing the difference with the year 1900 and the one returned by our API, but when they tried to process their “two-digit” year, which was now “100”, for display, they would take up one more byte than expected and end up corrupting whatever data was after it, which quickly led them to a crash.

And that was if you were lucky: some programs would appear to work correctly, but in fact have subtle yet devastating problems, such as computing interest backwards or outputting the wrong people ages.

We could not ignore the problem: starting about noon, 31st of December 1999 UTC, when the first parts of the world would start being in 2000, we would have been inundated with support requests for these defective products, never mind that the problem was not with us.

And we could not just block the faulty software: even if we did not already suspect that was the case, a survey showed every single one of our (important) customers was using at least one program which we know would exhibit issues come year 2000, with some customers using hundreds of such programs! And that’s without accounting for internally-developed software by the customer, and after requesting some sample we found out most of this software would be affected as well. Most of the problematic software was considered mission-critical and could not just be abandoned and had to keep working past 1999, come hell or high water.

Couldn’t the programs be fixed and customers get updated version? Well, for one in the usual case the company selling the program would be happy to do so, provided customers would pay for the upgrade to the updated version of the software, and customers reacted badly to that scenario.

And that assumes the company that developed the software was still in business.

In any case, the program might have been written in an obsolete programming language like Object Pascal, using the 16-bit APIs, and could no longer be built for lack of a surviving install of the compiler, or even lack of a machine able of running the compiler. Some of these programs could not be fixed without fixing the programming language they used or a library they relied on, repeating the problem recursively on the suppliers of these which may have become out of business. Even if the program could technically be rebuilt, maybe its original developer was long gone from the company and no one else could have managed to do it.

But a more common case was that the source code for the program was in fact simply lost to the ages.

Meanwhile, we were of course working on the solution. We came up with an elegant compatibility mechanism by which any application or other program which did not explicitly declare itself to support the years 2000 and after would get dates from the API in ETC instead of UTC. ETC was designed so that 1999 is the last year to ever happen. It simply never ends. You should really read the specification if you want the details, but basically how it works is that in the first half of 1999, one ETC second is worth two UTC seconds, so it can represent one UTC year; then in the first half of what is left of 1999, which is a quarter year, one ETC second is worth four UTC seconds, so again in total one UTC year, and in the first half of what is left after that, one ETC second is worth eight UTC seconds, etc. So we can fit an arbitrary number of UTC years into what seems to be one year in ETC, and therefore from the point of view of the legacy programs. Clever, huh? Of course, this means the resolution of legacy programs decreases as time goes on, but these programs only had a limited number of seconds they could ever account for in the future anyway, so it is making best use of the limited resource they have left. Things start becoming a bit more complex when we start dividing 1999 into amounts that are no longer integer amounts of seconds, but the general principle remains.

Of course, something might seem off in the preceding description, and you might guess that things did not exactly come to be that way. And indeed, when we deployed the solution in our usability labs, we quickly realized people would confuse ETC dates coming from legacy apps with UTC dates, for instance copying an ETC date and pasting it where a UTC date was expected, etc., causing the system to be unusable in practice. That was when we realized the folly of having two calendar systems in use at the same time. Something had to be done.

Oh, there was some resistance, of course. Some countries in particular dragged their feet. But in the end, when faced with the perspective of a digital apocalypse, everyone complied eventually, and by 1998 ETC was universally adopted as the basis for official timekeeping, just in time for it to be deployed. Because remember: application compatibility is paramount.

And besides, aren’t you glad it’s right now the 31st of December, 23:04:06.09375? Rather than whatever it would be right now had we kept “years”, which would be something in “2013” I guess, or another thing equally ridiculous.

Apple has a low-cost iPhone, but they should have a different one

From time to time rumors surface about Apple being poised to introduce some sort of cheap iPhone to fill a hole in their lineup, or we have so-called experts pontificate on why Apple needs to introduce such a device in order not to leave that market entirely to Android. The ways this gets discussed gets me wondering how these people could possibly not have noticed that Apple still sells the iPhone 4 and 4S, now as lower cost iPhones; I don’t know, maybe these don’t count because they have cooties? In reality, while they were introduced some time ago indeed, they can stand the comparison with “base” Android models as far as I can tell, and buyers do not seem to be snubbing them.

If at least the conversation was about whether the iPhone 4 and 4S were appropriate base or low-cost models for Apple to sell, as then there would be things to say on the matter (but no, it is always framed as if these two did not exist). Indeed, I think this strategy was justified when the iPhone 3G kept being sold along the new 3GS, it might have been tolerable to keep the iPhone 3GS along the new iPhone 4, but by now Apple should have long changed strategy. For a number of reasons, as the iPhone product lineup, or rather hardware platform, matures it should instead have low-cost models meant for that purpose.

The first reason goes back to the dark days of the Mac in the nineties, where former top of the line Macs would be discounted and sold as base models when another Mac model replaced them as the top of the line Mac; as a result, people would be hesitant to buy the (more) affordable Mac which they knew was not really up to date, and did not want to shell out for the current model, so they ended up just waiting for it to be discounted. It was hard for Apple to have a clear message on which Mac someone in a given market should be buying: so yesterday that model was not appropriate for consumers, but today it is? The heck? Fortunately, Steve Jobs put an end to that when he introduced the Mac product matrix (with its two dimensions: consumer-professional, and portable-desktop, and four models: iMac, PowerMac, iBook, and PowerBook).

Which brings me to the second reason, which is that not all technologies make sense as being introduced for professionals exclusively, even at first; USB, introduced in fact with the first iMac before it was on any PowerMac, is a prime example. Today, we have for instance SSDs, at least as an option or in the form of an hybrid drive.

But I think the most important reason is that having dedicated base models would allow Apple to sell devices where the hardware design flaws of yesterday (visible or invisible) are fixed, instead of having to keep having to take them into account in the next X years of software updates. While some new features in iOS releases have not been made available on older devices on which they run, the base OS has to run nevertheless, and even with this feature segmentation performance regressions have been observed. The other side of having specifically developed low-cost iPhone models is that Apple would seed with these devices current technologies to better be able to introduce new and interesting things in future versions of iOS (think, say, OpenCL), for instance because third-party developers are more likely to adopt a technology if they know every device sold in the last year supports it; this goes doubly if the technology cannot serve as an optional enhancement, but instead is meant to have apps depend on it.

The example the iPhone should follow, where Apple itself does it right, is with the iPad and in particular the iPad mini. I joke that now with the 128GB iPad there are 48 (count them) iPad SKUs, but that’s in fact OK, as the lineup is very orthogonal: from the consumer’s viewpoint, color and connectivity are similar to build-to-order options over capacity variations of the 3 base models; it must be somewhat challenging to manage supply and resupply, but apparently the operations at Apple is managing it; and sometimes the one combination you want is out of stock at the store, so you end up getting a slightly different one, but that’s minor in the grand scheme of things. On the other hand, the introduction of the iPad mini was indispensable to diversify the iPad presence and make the iPad feel like not just a single product, but a real hardware platform.

The one thing in particular that was done best in that respect with the iPad mini is that internally, the iPad mini is pretty much the iPad 2 guts with the 4th generation iPad connectivity: Lightning port, Bluetooth 4.0, and LTE as an option. I/O is one of these things where it often does not make sense to introduce technologies only at the high end at first, because of network effects: they apply whether the I/O is to connect devices with each other, in which case you need to clear a penetration threshold before people can use it in practice, or if the I/O is for accessories, in which case hardware accessory makers are more likely to follow the better you seed the technology to everyone.

Now I’m not saying it is going to be easy to do the same for the iPhone. It is clear that each iPhone hardware model is very integrated, requiring for each model a lot of investment not only in hardware design, but also in the supply chain and the assembling infrastructure. Designing each model to be sold for only one year would make it harder to amortize these investments, but planning for some hardware reuse during the design process could compensate to an extent, and I think the outcomes in having a clearer and stronger product lineup, better technology introduction, and decreased iOS maintenance costs, would make it worth it.

PSA: Subscribe to your own RSS feed

I like RSS. A lot. It allows me to efficiently follow the activity of a lot of blogs that do not necessarily update predictably (I mean, I’m guilty as charged here). So when things break down, in particular in silent or non-obvious ways, it is all the more grating. To avoid causing this for your readers, please follow the RSS feed of your own blog; it does not have to be much, you could do it for free with the built-in feature of Firefox, just do it.

Case in point: at some point Steven Frank was talking about the <canvas> tag (in a post since gone in one of his blog reboots). It showed up fine on the website, but for some reason I cannot fathom the angle brackets did not end up as being escaped in the XML feed, and NetNewsWire dutifully interpreted it as an actual canvas. With no closing tag, which means none of the text after the “tag” showed up in NetNewswire, so as far as I could tell the blog post just straight up ended there (before the tag) in a mysterious fashion. I fortunately thought that that didn’t look like Mr Frank’s usual style and investigated, but I might never have noticed anything wrong.

You might say it was because of a bug at some point in the feed generation, but this is not the point. I mean, this is the web, in permanent beta, of course bugs are going to happen. The point is that apparently the author never noticed it and fixed it, so he couldn’t possibly have been following his own feed; had he been doing so he would have noticed, and would have fixed it in any number of ways.

Another case is when the feed URL for Wil Shipley’s blog changed. He did not announce it anywhere, and I did not notice it until I went to his site at work and saw a post I had not seen before (and which had been posted earlier than that week). Had he been following his own feed, he would have noticed at some point when posting that the post never showed up in his reader, and would have remembered to notify his RSS subscribers in some way.

So kids, don’t be the lame guy: follow your own RSS feed. Otherwise, you’re publishing something you are not testing, and I think we recently talked about how bad that was. The more you know…

Proposal for a standard plain text format for iOS documents

Since the last times we visited the matter of working with documents on iOS, I have read with great interest more write-ups of people describing how they work on the iPad, because of course it is always good to hear about people being able to do more and more things on the iPad, but also because (as John Gruber so astutely noted) Dropbox almost always seems to be involved. I don’t feel that Dropbox solves the external infrastructure problem I raised in my first post on the matter, I consider Dropbox external infrastructure as well, if only because it requires you to be connected to the Internet to merely be able to transfer documents locally on your iPad (and that’s not a knock on Dropbox mind you, this is entirely the doing of restrictions Apple imposes).

I am going to concede one advantage the current de facto iOS model of documents in a per-app sandbox plus next to that an explicit container for document interchange, which is that it forces apps to actually consider support of interchange document formats. With the Grand Unified Model, and whatever we call the model Mac OS X now uses since Snow Leopard, applications would first only concern themselves with creating documents to save the state of the user’s work for them to pick it up later, without concern for other applications; and when the authors of the applications would come to consider standard formats, or at the very least creating an interchange format without data that are of no interest to another app (e.g. which tool was selected at the time the image was saved) or would amount to implementation details, they would realize that other applications had managed to slog through their undocumented document format to open it, and as a result the authors did not feel so pressured to support writing another document format. The outcome is that the onus of information interchange falls only on the readers, which need to keep adding support for anything the application that writes the document feels like adding to the format, in the way it feels like doing so.

However, with the de facto model used by iOS, apps may start out the same way, but when they want to claim Dropbox support, they have damn well better write in there documents in a standard or documented interchange format, or their claims of Dropbox support become pretty much meaningless. I am not sure the tradeoff is worth it compared to the loss of being able to get at the original document directly as a last resort (in case, for instance, the document exchanged on Dropbox has information missing compared to the document kept in the sandbox), but it is indeed an advantage to consider. An issue with that, though, is that as things currently stand there is no one to even provide recommendations as to the standard formats to use for exchanging documents on iOS: Dropbox the company is not in a position to do so, and as far as Apple is concerned document interchange between iOS apps does not exist.

So when I read that the format often used in Dropbox to exchange data between apps is plain text, while this is better than proprietary formats, this saddens me to no end. Why? Because plain text is a lie. There is no such thing as plain text. Plain text is a myth created by Unix guys to control us. Plain text is a tall tale parents tell their children. Plain text is what you find in the pot at the end of the rainbow. Plain text is involved in Hercules’ labors and Ulysses’ odyssey. Perpetual motion machines run on plain text. The Ultimate Question of Life, the Universe and Everything is written in plain text.

I sense you’re skeptical, so let me explain. Plain text is pretty well defined, right? ASCII, right? Well, let me ask you: what is a tab character supposed to do? Bring you over to the next tab stop every 4 spaces? Except that on the Mac tab stops are considered to occur every 8 spaces instead (and even on Unix not everyone agrees). And since we are dealing with so-called plain text, the benefit of being able to align in a proportional context does not apply: if you were to rely on that, then switch to another editor that uses a different font, or switch the editing font in your editor, then your carefully aligned document would become all out of whack. Finally, any memory saving brought by the tab character has become insignificant given today’s RAM and storage capacities.

Next are newlines. Turns out, hey, no one agrees here either: you’ve got carriage return, line feed, the two together (and both ways). More subtle is wrapping… What’s this, you say? Editors always word wrap? Except piconano, for instance, doesn’t by default. And Emacs, in fact, does a character wrap: by default it will cut in the middle of a word. It seems inconsequential, but it causes users of non-wrapping editors to complain that others send them documents with overly long lines, while these others complain that the first guys write lines with an arbitrary limit, causing for instance unsightly double-wrapping when used on a window narrower than that arbitrary width.

And of course, you saw it coming, comes the character encoding, we have left the 7-bit ASCII world eons ago. Everything is Unicode capable by now, but some idiosyncrasies still remain: for instance as far as I can tell out of the box TextEdit in Mac OS X still opens text files in MacRoman by default.

This is, simply, a mess. There is not one, but many plain text formats. So what can we do?

The proposal

Goals, scope and rationale (non-normative)

The most important, defining characteristic of the proposal for Standard Plain Text is that it is meant to store prose (or poetry). Period. People might sometimes happen to use it for, e.g. source code, but these use cases shall not be taken into considerations for the format. If you want to edit makefiles or a tab separated values file, use a specialized tool. However, we do want to make sure that more specialized humane markup/prose-like formats can be built above the proposal, in fact for instance Markdown and Textile over Standard Plain Text ought to be able to be trivially defined as being, well, Markdown and Textile over Standard Plain Text.

Then, we want to be able to recover the data on any current computer system in case of disaster. This means compatibility with existing operating systems, or at least being capable of recovering the data using only programs built in these operating systems.

And we want the format to be defined thoroughly enough to limit as much as possible disagreements and misunderstandings, while keeping it simple to limit risks of mistakes in implementations.

Requirements

Standard Plain Text files shall use the Unicode character set, and be encoded in UTF-8. Any other character set or encoding is explicitly forbidden.

This seem obvious, until you realize this causes Asian text, among others, to take up 50% more storage than it would using UTF-16, so there is in fact a tradeoff here, and compatibility was favored; I apologize to all our Japanese, Chinese, Korean, Indian, etc. friends.

Standard Plain Text files shall not contain any character in the U+0000 – U+001F range, inclusive, (ASCII control characters) except for U+000A (LINE FEED). As a result, tabulation characters are forbidden and the line ending shall be a single LINE FEED. Standard Plain Text files shall not contain any character in the U+00FF – U+011F range, inclusive (DELETE and C1 range). Standard Plain Text files shall not contain any U+FEFF character (ZERO WIDTH NO-BREAK SPACE aka byte order mark), either at the start or anywhere else. All other code points between U+0020 and U+10FFFF, inclusive, that are allowed in Unicode are allowed, including as-yet unassigned ones.

Standard Plain Text editors shall word wrap, and shall support arbitrarily long stretches of characters and bytes between two consecutive LINE FEEDs. They may support proportional text, but they shall support at least one monospace font.

These requirements shall be enforced at multiple levels in Standard Plain Text editors, both at the user input stage and when writing to disk at least: pasting in text containing forbidden characters shall not result in them being written as part of a Standard Plain Text file. Editors may handle tabulation given as input any way they see fit (e.g. inserting N spaces, inserting enough spaces to reach the next multiple of N column, etc.) as long as it does not result in a tab character being written as part of a Standard Plain Text file in any circumstance.

Standard Plain Text files should have the .txt extension for compatibility. No MacOS 4-char type code is specified. No MIME type is specified for the time being. If a Uniform Type Identifier is desired, net.wanderingcoder.projects.standard-plain-text (conforming to public.utf8-plain-text) can be used as a temporary solution.

Clearly this is the part that still needs work. Dropbox supports file metadata, but I have not fully investigated the supported metadata, in particular whether there is a space for an UTI.

Appendix A (non-normative): recovery methods

On a modern Unix/Linux system: make sure the locale is a UTF-8 variant, then open with the text editor of your preference.

On a Mac OS X system: in the open dialog of TextEdit, make sure the encoding is set to Unicode (UTF-8), then open the file. Being a modern Unix, the previous method can also be applied.

On a modern Windows system (Windows XP and later): in the open dialog of Wordpad, make sure the format to open is set to Text Document (.txt) (not Unicode Text Document (.txt)), then open the file. Append a newline then delete it, then save the file. In the open dialog of Notepad, make sure the encoding is set to UTF-8, then open the latter file.

The reason for this roundabout method is that Wordpad does not support UTF-8 (Unicode in its opening options in fact means UTF-16) but supports linefeed line endings, while Notepad does not support linefeed line endings. Tested on Windows XP.

Creating discoverable user defaults

John C. Welch recently lashed out at Firefox and Chrome for using non-standard means of storing user settings, and conversely praised Safari for using the property list format, the standard on Mac OS X, to do so. A particular point of contention with the Chrome way was the fact there is no way to change plug-in settings by manipulating the preferences file, unless the user changed at least one such setting in the application beforehand, while this is not the case with Safari.

Or is it? As mentioned in the comments, there is at least one preference in Safari that behaves in a similar way, remaining hidden and absent from the preferences file until explicitly changed. What gives?

In this post I am going to try and explain what is going on here, and provide a way for Mac (and iOS — you never know when that might turn out to be useful) application developers to allow the user defaults that their applications use to be discoverable and changeable in the application property list preferences file.

The first thing is that it is an essential feature of the Mac OS X user defaults system (of which the property list format preference files are a part) that preferences files need not contain every single setting the application relies upon: if an application asks the user defaults system for a setting which is absent from the preference plist for this application, the user default system will merely answer that there no value for that setting, and the application can then take an appropriate action. This is essential because it allows, for instance, to seamlessly update apps while preserving their settings, even when the update features new settings, but also to just as seamlessly downgrade such apps, and even to switch versions in other ways: suppose you were using the Mac App Store version of an app, and you switch to the non-Mac App Store one, which uses Sparkle for updates; using the Mac OS X user defaults system, the non-Mac App Store version will seamlessly pick up the user settings, while when Sparkle will ask for its settings (it has a few) in the application preferences it will find they are not set, and Sparkle will act as if there was no Sparkle activity previously for this application, in other words behave as expected. This is much more flexible than, for instance, a script to upgrade the preference file at each app update.

So, OK, the preference file need not contain every single setting the application relies upon before the application is run; what about after the application is run? By default, unfortunately, the Mac OS X user defaults system is not made aware of the decision taken by the application when the latter gets an empty value, so the user defaults system simply leaves the setting unset, and the preference file remains as-is. There is, however, a mechanism by which the application can declare to the user defaults system the value to use when the search for a given setting turns out empty, by registering them in the registration domain; but as it turns out, these default settings do not get written in the preferences file either, they simply get used as the default for each setting.

So in other words, when using techniques recommended by Apple, no setting will ever get written out to the preferences file until explicitly changed by the user. It is still possible to change the setting without using the application, but this requires knowing exactly the name of the setting, its location, and the kind of value it can take, which are hard to guess without it being present in the preference file. This will not do, so what can we do?

What I have done so far in my (not widely released) applications is the following (for my fellow Mac developers lucky enough to be maintaining Carbon applications, adapt using CFPreferences as appropriate): to begin with I always use -[NSUserDefaults objectForKey:] rather than the convenience methods like -[NSUserDefaults integerForKey:], at the very least as the first call, so that I can known whether the value is unset or actually set to 0 (which is impossible to tell with -[NSUserDefaults integerForKey:]); then if the setting was not found in the user defaults system, I explicitly write the default value to the user defaults system before returning it as the asked setting; for convenience I wrap the whole thing into a function, one for each setting, looking more or less like this:

NSString* StuffWeServe(void)
{
   NSString* result = nil;
   
   result = [[NSUserDefaults standardUserDefaults] objectForKey:@"KindOfObjectWeServe"];
   if (result != nil) // elided: checks that result actually is a string
      return result;
   
   // not found, insert the default
   result = @"burgers";
   [[NSUserDefaults standardUserDefaults] setObject:result forKey:@"KindOfObjectWeServe"];
   
   return result;
}

It is important to never directly call NSUserDefaults and always go through the function whenever the setting value is read (writing the setting can be done directly through NSUserDefaults). I only needed to do this for a handful of settings, if you have many settings it should be possible to implement this in a systemic fashion by subclassing NSUserDefaults and overriding objectForKey:, to avoid writing a bunch of similar functions.

Using this code, after a new setting is requested for the first time it is enough to synchronize the user defaults (which should happen automatically during the application run loop) for it to appear in the preference file and so more easily allow it to be discovered and changed by end users, or more typically, system administrators.

Meta-Patents

My apologies for the extended radio silence prior to Patents; it took me a lot of time to properly articulate my reasoning, as it is no simple matter, and in particular it took me time to properly show in which ways software differs from other engineering domain and why it matters for patents. The blog post I ended up writing is admittedly a bit “too long, didn’t read”, but given the often simplistic discourse occurring these days around software patents, I wanted to start back from the basics so it necessarily got a bit long.

In other news, as promised I updated my existing posts, namely here Introduction to NEON on iPhone and A few things iOS developers ought to know about the ARM architecture, to account for ARMv7s. However an equivalent of Benefits (and drawback) to compiling your iOS app for ARMv7 for ARMv7s will have to wait, as I do not have an ARMv7s device I can call mine, so I have no idea how performance improves in practice when targeting ARMv7s. In the meantime I advise measuring the improvements in the specific case of your apps to see whether it is worth it, I cannot provide any general advice yet.

Patents

Patents and their application to software have been in the news lately: Lodsys and other entities that seem to have been created from whole cloth for that sole purpose are suing various software companies for patent infringement, Android is attacked (directly or indirectly) by historical operating system actors Apple, Microsoft and Oracle (as owner of Sun) for infringing their patents, web video is standardized but the codec is left unspecified as the W3C will only standardize on freely-licensable technologies while any remotely modern video compression technique is patented (even the ostensibly patent-free WebM codec is heading towards having a patent pool formed around it).

Many in the software industry consider it obvious that not only a reform is needed, but that software patents should be banned entirely given their unacceptable effects; however I haven’t seen much of a justification of why they should be banned, as often the article/blog post/editorial defending this position considers it obvious. Well, it is certainly obvious for the author as a practitioner of software, and obvious to me as the same, but it’s not to others, and I wouldn’t want engineers of other trades to see software developers as prima donnas who think they should be exempted from the obligations related to patents for no reason other than the fact it inconveniences them. So here I am going to expose why I consider that software patents actually discourage innovation, and in fact discourage any activity, in the software industry.

Why the current situation is untenable

Let’s start by the basics. A patent is an invention that an inventor, in exchange for registering it in a public office (which includes a fee), is given exclusive rights to. Of course, he can share that right by licensing the patent to others, or he can sell the patent altogether. Anyone else using the invention (and that includes an end user) is said to be infringing the patent and is in the wrong, even if he came up with it independently. That seems quite outlandish, but it’s a tradeoff that we as a society have made: we penalize parallel inventors who are of good faith in order to better protect the original inventor (e.g. to avoid copyists getting away with their copying by pretending they were unaware of the original invention). Of course, if the parallel inventor is not found to have been aware of the original patent, he is less penalized than if he were, but he is penalized nonetheless. The aim is to give practitioners in a given domain an incentive to keep abreast of the state of the art in various ways, including by reading the patents published by the patent office in their domain. In fields where the conditions are right, I hear it works pretty well.

And it is here we see the first issue with software patents: the notorious incompetence of the USPTO (United States Patent and Trademark Office)1, which has been very lax and inconsistent when it comes to software patents, and has granted a number of dubious ones; and I hear it’s not much better in other countries where software patents are granted (European countries thankfully do not grant patents on software, for the most part). One of the criteria when deciding whether an invention can be patented is whether it is obvious to a practitioner aware of the state of the art, and for reasonably competent software developers the patents at the center of some lawsuits are downright obvious inventions. The result is that staying current with the software patents that are granted is such a waste of time that it would sink most software companies faster than any patent suit.

Now, it is entirely possible that the USPTO is overworked with a flood of patent claims which they’re doing their best to evaluate given their means, and the bogus patents that end up being granted are rare exceptions. I personally believe the ones we’ve seen so far are but the tip of the iceberg (most are probably resting along with more valid patents in the patent portfolios of big companies), but even if we accept they are an exception, it doesn’t matter because of a compounding issue with software patents: litigation is very expensive. To be more specific, the U.S. patent litigation system seems calibrated for traditional brick and mortar companies that produce physical goods at the industrial scale; calibrated in the sense of how much scrutiny is given to the patents and the potential infringement, the number of technicalities that have to be dealt with before the court gets to the core of the matter, how long the various stages of litigation last, etc. Remember that in the meantime, the lawyers and patent attorneys gotta get paid. What are expensive but sustainable litigation expenses for these companies simply put most software companies, which operate at a smaller scale, out of business.

Worse yet, even getting to the point where the patent and the infringement are looked at seriously is too expensive for most companies. As a result, attackers only need to have the beginning of a case to start threatening software developers with a patent infringement lawsuit if they don’t take a license; it doesn’t matter if the attacker’s case is weak and likely to lose in court eventually, as these attackers know that the companies they’re threatening do not have the means to fight to get to that point. And there is no provision for the loser to have to pay for the legal fees of the winner. So the choice for these companies is either to back off and pay up, or spend at least an arm and a leg that they will never recover defending themselves. This is extortion, plain and simple.

So even if bogus patents are the exception, it is more than enough for a few of them to end up in the wild and used as bludgeons to waylay software companies pretty much at will, so the impact is disproportionate with the number of bogus patents. Especially when you consider the assailants cannot be targeted back since they do not produce products.

But at the very least, these issues appear to be fixable. The patent litigation system could be scaled back (possibly only for software patents), and, who knows, the USPTO could change and do a correct job of evaluating software patents, especially if there are disincentives in place (like a higher patent submission fee) to curb the number of submissions and allow the USPTO to do a better job. And one could even postulate a world where software developers “get with the program” and follow patent activity and avoid patented techniques (or license them, as appropriate) such that software development is no longer a minefield. But I am convinced this will not work, especially the latter, and that software (with possible exceptions) should not be patentable, for reasons I am going to expose.

Why software patents themselves are not sustainable

The first reason is that contrary to, say, mechanical engineers, or biologists, or even chip designers, the software development community is dispersed, heterogeneous, and loosely connected, if at all. An employee in IT writing network management scripts is a software practitioner; an iOS application developer is a software practitioner; a web front-end developer writing HTML and JavaScript is a software practitioner; a Java programmer writing line of business applications internal to the company is a software practitioner; an embedded programmer writing the control program for a washing machine is a software practitioner; a video game designer scripting a dialog tree is a software practitioner; a Linux kernel programmer is a software practitioner; an embedded programmer writing critical avionics software is a software practitioner; an expert writing weather simulation algorithms is a software practitioner; a security researcher writing cryptographic algorithms is a software practitioner. More significantly, every company past a certain size, regardless of its field, will employ software practitioners, if only in IT, and probably to write internal software related to its field. Software development is not limited to companies in one or a few fields, software practitioners are employed by companies from all industry and non-industry sectors. So I don’t see software developers ever getting into a coherent enough “community” for patents to work as intended.

The second reason, which compounds the first, is that software patents can not be reliably indexed, contrary to, say, chemical patents used in the pharmaceutical industry for instance2. If an engineer working in pharmacology wants to know whether the molecule he intends to work on is patented already, there are databases that, based on the formal description of the molecule, allow to find any and all patents covering that molecule, or allow the knowledge with a reasonably high degree of confidence that the molecule is not patented yet if the search turns up no result. No such thing exists (and likely no such thing can exist) for software patents, where there is at best keyword search; this is less accurate, but in particular cannot give confidence that an algorithm we want to clear is not patented, as a keyword search may miss patents that would apply. It appears that the only way to ensure a piece of software does not infringe patents is to read all software patents (every single one!) as they are issued to see if one of them wouldn’t cover the piece of software we want to clear; given that every company that produces software would need to do so, and remember the compounding factor that this includes every company past a certain size, this raises some scalability challenges, to put it lightly.

This is itself compounded by the fact you do not need a lot of resources available, or to spend a lot of resources or time, to develop and validate a software invention. To figure out whether a drug is worth patenting (to say nothing of producing it in the first place), you need a lab, in which you run experiments taking time and money to pay for the biological materials, the qualified technicians tending to the experiments, etc. Which may not work, in which case you have to start over; one success has to bear the cost of likely a magnitude more failures. To figure out whether a mechanical invention is worth patenting, you need to build it, spend a lot of materials (ones constitutive of the machine because it broke catastrophically, or ones the machine is supposed to process like wood or plastic granules) iterating on the invention until it runs, and even then it may not pan out in the end. But validating a software invention only requires running it on a computer that can be had for $500, eating a handful of kilojoules (kilojoules! Not kWhs, kilojoules, or said another way, kilowatt-seconds) of electrical power, and no worker time at all except waiting for the outcome, since everything in running software is automated. With current hardware and compilers, the outcome of whether a software invention works or not can be had in mere seconds, so there is little cost to failure of an invention. As a result, developing a software invention comparable in complexity to an invention described in a non-software patent has a much, much lower barrier of entry and requires multiple orders of magnitude less resources; everyone can be a software inventor. Now there is still of course the patent filing fee, but still in software you’ve got inventions that are easier to come up with, as a result many more of them will be filed, while they impact many more companies… Hmm…

Of course, don’t get me wrong, I do not mean here that software development is easy or cheap, because software development is about creating products, not inventions per se; developing a product involves a lot more (like user interface design, getting code by different people to run together, figuring out how the product should behave and what users want in the product, and many other things, to say nothing of non-programming work like art assets, etc.) than simply the inventions contained inside, and developing that takes a lot of time and resources.

Now let us add the possibility of a company getting a software patent so universal and unavoidable that the company is thus granted a monopoly on a whole class of software. This has historically happened in other domains, perhaps most famously with Xerox who for long held a monopoly on copying machines, by having the patent on the only viable technique for doing so at the time. But granting Xerox a monopoly on the only viable copying technique did not impact other markets, as this invention was unavoidable for making copying machines and… well, maybe integrated fax/printer/copying machine gizmos which are not much more than the sum of their parts, but that was it. On the other hand, a software invention is always a building block for more complex software, so an algorithmic patent could have an unpredictable reach. Let us take the hash table, for instance. It is a container that allows to quickly (in a sense formally defined) determine whether it already contains an object with a given name, and where, while still allowing to quickly add a new object; something computer memories by themselves are not able to do. Its performance advantages do not merely make programs that use it faster, they allow many programs, which otherwise would be unfeasibly slow, to exist. The hash table enables a staggering amount of software; for instance using a hash table you can figure out in a reasonable time from survey results the list of different answers given in a free-form field of that survey, and for each such answer the average age of respondents who picked that answer (as an example). Most often the various hash tables uses are even further removed from user functionality, but are no less useful, each one providing its services to another software component which itself provides services to another, etc. in order to provide the desired user functionality. Thanks to the universal and infinitely composable nature of software there is no telling where else, in the immensity of software, a software invention could be useful.

Back when it was invented, the hash table was hardly obvious, had it been patented everyone would have had to find alternative ways to accomplish more or less the same purpose (given the universal usefulness it has), such as trees, but those would themselves have become patented until they was no solution left, as there are only so many ways to accomplish that goal (given that in software development you cannot endlessly vary materials, chemical formulas, or environmental conditions); at that point software development would have become frozen in an oligopoly of patent-having companies, which would have taken advantage of being the only ones able to develop software to file more patents to indefinitely maintain that advantage.

Even today, software development is still very young compared to other engineering fields, even to what they were around the start of the nineteenth century when patent systems were introduced. And its fundamentals, such as the hardware it runs on and its capabilities, change all the time, such that there is always a need to reinvent some of its building blocks; therefore patenting techniques currently being developed risks having enormous impact on future software.

But what if algorithmic inventions that are not complex by software standards were not allowed patent protection, and only complex (by software standards) algorithms were, to compensate for the relative cheapness of developing an algorithmic invention of complexity comparable to a non-algorithmic invention, and avoid the issue of simple inventions with too important a reach? The issue is, with rare exceptions complex software does not constitute an invention bigger than the sum of individual inventions. Indeed, complex software is developed to solve user needs, which are not one big technical problem, but rather a collection of technical problems the software needs to solve, such that the complex software is more than the sum of its parts only to the extent these parts work together to solve a more broadly defined, non-technical problem (that is, the user needs). However this complex software is not a more complex invention solving a new technical problem its individual inventions do not already solve, so patenting this complex software would be pointless.

Exceptions (if they are possible)

This does leave open the possibility of some algorithmic techniques for which I would support making an exception and allowing them patent protection while denying it to algorithms in general, contingent on a caveat I will get into afterwards.

First of these are audio and video compression techniques: while they come down to algorithms in the end, they operate on real world data (music, live action footage, voice, etc.) and have shown to be efficient at compressing this real-world data, so they have more than just mathematical properties. But more importantly, these techniques compress data by discarding information that will end up not being noticed as missing by the consumer of the media once uncompressed, and this has to be determined by experimentation, trial and error, large user trials, etc. that take resources comparable to a non-algorithmic invention. As a result, the economics of developing these techniques is not at all similar to software, and application of these techniques is bounded to some, and not all, software applications, so it is worth considering keeping patent protection for these techniques.

Other techniques which are worth, in my opinion, patenting even though they are mostly implemented in software are some encryption/security systems. I am not necessarily talking here of encryption building blocks like AES or SHA, but rather of setups such as PGP. Indeed these setups have provable properties as a whole, so they are more than just the sum of their parts; furthermore, as with all security software the validation that such techniques work can not be done by merely running the code3, but only by proving (a non-trivial job) that they are secure, again bringing the economics more in line with those of non-algorithm patents, therefore having these techniques in the patent system should be beneficial.

So it could be worthwhile to try and carve an exception and allow patents for these techniques and others sharing the same patent-system-friendly characteristics, but if attempted extreme care will have to be taken when specifying such an exception. Indeed, even in the U.S.A. algorithm patents are formally banned, but accumulated litigations ended up with court decisions that progressively eroded this ban, first allowing algorithms on condition they were intimately connected to some physical process, then easing more and more that qualification until it became meaningless; software patents must still pretend being about something other than software or algorithms, typically being titled some variation of “method and apparatus”, but in practice the ban on algorithm patents is well and truly gone, having been loopholed to death. So it is a safe bet any granted exception, on an otherwise general ban on software patents should it happen in the future, will be subject to numerous attempts to exploit it for loopholes to allow software in general to be patented again, especially given the important pressure from big software companies to keep software patents valid.

So if there is any doubt as to the validity and solidity of a proposed exception to a general ban on software patents, then it is better to avoid general software patents coming back through a back door, and therefore better to forego the exception. Sometimes we can’t have nice things.

Other Proposals

Nilay Patel argues that software patents should be allowed, officially even. He mentions mechanical systems and a tap patent in particular, arguing that since the system can be entirely modeled using physical equations, fluid mechanics in particular, the entire invention comes down to math in the end like for software, so why should software patents be treated differently and banned? But the key difference here, to take again the example of the tap patent he mention, is that the part of math which is an external constraint, the fluid mechanics, are an immutable constant of nature. On the other hand with algorithm patents all algorithms involved are the work of man; even if there are external constraining algorithms in a given patent, due to legacy constraints for instance, these were the work of man too. In fact, granting a patent because an invention is remarkable due to the legacy constraints it has to work with and how it solves them would indirectly encourage the development and diffusion of such constraining legacy! We certainly don’t want the patent system encouraging that.

The EFF proposes, among other things, allowing independent invention as a valid defense again software patent infringement liabilities. If this is allowed, we might as well save costs and abolish software patents in the first place: a patent system relies on independent infringement being an infringement nonetheless in order to avoid abuses rendering the whole system meaningless, and I do not see software being any different in that regard.

I cannot remember where, but I heard the idea, especially with regard to media compression patents, of allowing software implementations to use patented algorithm inventions without infringing, so that software publishers would not have to get a license, while hardware implementations would require getting a license. But an issue is that “hardware” implementations are sometimes in fact DSPs which run code actually implementing the codec algorithms, so with this scheme the invention could be argued to be implemented in software; therefore OEMs would just have to switch to such a scheme if they weren’t already, qualify the implementation as software, and not have to pay for any license, so it would be equivalent to abolishing algorithm patents entirely.


  1. I do not comment on the internal affairs of foreign countries in this blog, but I have to make an exception in the case of the software patent situation in the U.S.A., which is so appalling that it ought to be considered a trade impediment.

  2. I learned that indexability was a very useful property that, in contrast to software patents, some patent domains did have, and the specific example of the pharmaceutical industry as such a domain, from an article on the web which I unfortunately cannot find at the moment; a search on the web did not allow me to find it but turned up other references for this fact.

  3. It’s like a lock: you do not determine that a lock you built is fit for fulfilling its purpose by checking that it closes and that using the key opens it; you determine it by making sure there is no other way to open it.

PSA: Do not release ARMv7s code until you have tested it

If you are using a third-party SDK in your iOS app, you may encounter a problem when linking with the current Xcode release: in that case the linker errors with the following line in the build log:

ld: file is universal (2 slices) but does not contain a(n) armv7s slice: libexample.a for architecture armv7s

(with some libraries, the linker will instead output the following when it errors, but it’s the same general problem:)

ld: warning: ignoring file libexample.a, file was built for archive which is not the architecture being linked (armv7s): libexample.a
Undefined symbols for architecture armv7s:

One solution for this issue is to get an updated version of the SDK that has a library with an ARMv7s slice (provided such an update exists, of course, otherwise you have no choice but to apply the second solution). However you should do so only if you have an iPhone 5 to test on; otherwise, I strongly recommend you apply the second solution: go to your project settings, (or target settings, if they are overridden at the target level), and edit the Architectures setting from “armv7 armv7s” to just “armv7”.

Why? Well, only the iPhone 5 can run the variant of your app code (called a slice) compiled for ARMv7s, so if you build and eventually release an update to your app that includes ARMv7s support, you would be releasing code you have not tested yourself, which is a big no-no. Don’t do it. In fact even if you can test on an iPhone 5, there is likely no need to rush and add ARMv7s support in your app as the benefits are incremental at best, as far as I can tell (but do not take my word for it, measure!); I really can’t understand why Apple added ARMv7s support in such a way that existing projects start using it right away by default.

Look forward to a post detailing the benefits (if any) to adding an ARMv7s slice to your app, as well as updates to my existing posts, in the coming weeks.

This post was initially published with somewhat different contents as at the time the iPhone 5 was not actually available. In the interest of historical preservation, the original content has been moved here.

New section: comicroll

The absence of a blogroll on this site is very much intentional. For me the main issue with a blogroll is that it provides no context, it does not tell why the author of the blog reads the linked blogs, and more importantly does not tell why the author of the blog thinks his audience would benefit from reading the linked blogs. For instance, it might surprise a few of you to learn that I enjoy reading the Old New Thing, Raymond Chen’s blog on the the history of Windows and Win32 arcana. A great French humorist once said: “Are you going to remain simplistic (literally, primary) anticommunists your whole life, while it is enough to read Marx to become straight away thoughtful (literally, secondary) anticommunists?” I have not read The Capital, but I read the Old New Thing for a similar purpose. This is very important context that you would not have if I simply were to put a link to the Old New Thing in a blogroll.

When I link to other blogs in one of my posts, on the other hand, it does provide you with the context to help you decide if you want to read that post, and potentially if you want to read more of that blog. However, there exists content that most of the time is not practical to drop a link to in a blog post, such as webcomics. This is why I am pleased to introduce a comicroll to this site. I religiously follow all works listed in this comicroll, never missing an update, and I recommend them all heartily. For the benefit of those of you following at home on an RSS reader, they are, at the time of this writing:

Note that a few of them are in fact over or in hiatus, but they are too good to pass up. In particular, the Daily Victim (which by the way is not exactly a comic, but rather illustrated humor prose) was a GameSpy feature that used to be available here, and this is in fact an archive saved by a fellow fan and made available on the web (by the way, if he happens to be reading this, I would like to contact him about some of the images in this archive Never mind, the issue has been fixed, it’s getting better all the time).

So please enjoy this comicroll; know that it is not going to be set in stone, I will be changing it now and again, so be sure to check it out from time to time.