All posts in category Programming

How does ETC work? A sad story of application compatibility

This post is, in fact, not quite like the others. It is a parody of the Old New Thing I wrote for parody week, so take it with a big grain of salt…

Commenter Contoso asked: “What’s the deal with ETC? Why is it so complicated?“

First, I will note that ETC (which stands for Coordinated Eternal Time) is in fact an international standard, having been adopted by ISO as well as national and industrial standard bodies. The specification is also documented on MSDN, but that’s more for historical reasons than anything else at this point, really. But okay, let’s discuss ETC, seeing as that’s what you want me to do.

ETC is not complicated at all if you follow from the problem to its logical conclusion. The youngest among you might not realize it, but the year 2000 bug was a Big Deal. When it began to be discussed in the public sphere starting in 1996 or so, most people laughed it off, but we knew, and always knew that if nothing was done computers, and all civilization in fact, would be headed for a disaster of biblical proportions. Real wrath of God type stuff. The dead rising from the grave! Human sacrifice! Dogs and cats living together… mass hysteria!

The problem originated years before that when some bright software developers could not be bothered to keep track of the whole year, and instead only kept track of the last two digits; so for instance, 1996 would be stored as just 96 in memory, and when reading it it was implicitly considered to have had the “19” before it, and so would be restored as “1996” for display, processing, etc. But it just happened to work because the years they saw started in “19”, and things would go wrong as soon as years would no longer do so, starting with 2000.

What happened (or rather, would have happened if we let it happen, this was run in controlled experiment conditions in our labs) in this case was that, for starters, these programs would print the year in the date as “19100”. You might think that would not be too bad, even though that would have some regulatory and other consequences, and would result in customers blaming us, and not the faulty program.

But that would in fact be if they even got as far as printing the date.

Most of them just fell over and died from some “impossible” situation long before that: some would take the date given by the API, convert it to text, blindly take the last two digits without checking the first two, and when comparing with the date in its records to see how old the last save was would end up with a negative age since it did 0 – 99 as far as the year was concerned, and the program would crash on a logic error; others would try and behave better by computing the difference with the year 1900 and the one returned by our API, but when they tried to process their “two-digit” year, which was now “100”, for display, they would take up one more byte than expected and end up corrupting whatever data was after it, which quickly led them to a crash.

And that was if you were lucky: some programs would appear to work correctly, but in fact have subtle yet devastating problems, such as computing interest backwards or outputting the wrong people ages.

We could not ignore the problem: starting about noon, 31st of December 1999 UTC, when the first parts of the world would start being in 2000, we would have been inundated with support requests for these defective products, never mind that the problem was not with us.

And we could not just block the faulty software: even if we did not already suspect that was the case, a survey showed every single one of our (important) customers was using at least one program which we know would exhibit issues come year 2000, with some customers using hundreds of such programs! And that’s without accounting for internally-developed software by the customer, and after requesting some sample we found out most of this software would be affected as well. Most of the problematic software was considered mission-critical and could not just be abandoned and had to keep working past 1999, come hell or high water.

Couldn’t the programs be fixed and customers get updated version? Well, for one in the usual case the company selling the program would be happy to do so, provided customers would pay for the upgrade to the updated version of the software, and customers reacted badly to that scenario.

And that assumes the company that developed the software was still in business.

In any case, the program might have been written in an obsolete programming language like Object Pascal, using the 16-bit APIs, and could no longer be built for lack of a surviving install of the compiler, or even lack of a machine able of running the compiler. Some of these programs could not be fixed without fixing the programming language they used or a library they relied on, repeating the problem recursively on the suppliers of these which may have become out of business. Even if the program could technically be rebuilt, maybe its original developer was long gone from the company and no one else could have managed to do it.

But a more common case was that the source code for the program was in fact simply lost to the ages.

Meanwhile, we were of course working on the solution. We came up with an elegant compatibility mechanism by which any application or other program which did not explicitly declare itself to support the years 2000 and after would get dates from the API in ETC instead of UTC. ETC was designed so that 1999 is the last year to ever happen. It simply never ends. You should really read the specification if you want the details, but basically how it works is that in the first half of 1999, one ETC second is worth two UTC seconds, so it can represent one UTC year; then in the first half of what is left of 1999, which is a quarter year, one ETC second is worth four UTC seconds, so again in total one UTC year, and in the first half of what is left after that, one ETC second is worth eight UTC seconds, etc. So we can fit an arbitrary number of UTC years into what seems to be one year in ETC, and therefore from the point of view of the legacy programs. Clever, huh? Of course, this means the resolution of legacy programs decreases as time goes on, but these programs only had a limited number of seconds they could ever account for in the future anyway, so it is making best use of the limited resource they have left. Things start becoming a bit more complex when we start dividing 1999 into amounts that are no longer integer amounts of seconds, but the general principle remains.

Of course, something might seem off in the preceding description, and you might guess that things did not exactly come to be that way. And indeed, when we deployed the solution in our usability labs, we quickly realized people would confuse ETC dates coming from legacy apps with UTC dates, for instance copying an ETC date and pasting it where a UTC date was expected, etc., causing the system to be unusable in practice. That was when we realized the folly of having two calendar systems in use at the same time. Something had to be done.

Oh, there was some resistance, of course. Some countries in particular dragged their feet. But in the end, when faced with the perspective of a digital apocalypse, everyone complied eventually, and by 1998 ETC was universally adopted as the basis for official timekeeping, just in time for it to be deployed. Because remember: application compatibility is paramount.

And besides, aren’t you glad it’s right now the 31st of December, 23:04:06.09375? Rather than whatever it would be right now had we kept “years”, which would be something in “2013” I guess, or another thing equally ridiculous.

by Pierre Lebeaupin on April 1, 2013  •  Permalink
Posted in Parody, Programming, Technology

Posted by Pierre Lebeaupin on April 1, 2013

http://wanderingcoder.net/2013/04/01/9917052/

Creating discoverable user defaults

John C. Welch recently lashed out at Firefox and Chrome for using non-standard means of storing user settings, and conversely praised Safari for using the property list format, the standard on Mac OS X, to do so. A particular point of contention with the Chrome way was the fact there is no way to change plug-in settings by manipulating the preferences file, unless the user changed at least one such setting in the application beforehand, while this is not the case with Safari.

Or is it? As mentioned in the comments, there is at least one preference in Safari that behaves in a similar way, remaining hidden and absent from the preferences file until explicitly changed. What gives?

In this post I am going to try and explain what is going on here, and provide a way for Mac (and iOS — you never know when that might turn out to be useful) application developers to allow the user defaults that their applications use to be discoverable and changeable in the application property list preferences file.

The first thing is that it is an essential feature of the Mac OS X user defaults system (of which the property list format preference files are a part) that preferences files need not contain every single setting the application relies upon: if an application asks the user defaults system for a setting which is absent from the preference plist for this application, the user default system will merely answer that there no value for that setting, and the application can then take an appropriate action. This is essential because it allows, for instance, to seamlessly update apps while preserving their settings, even when the update features new settings, but also to just as seamlessly downgrade such apps, and even to switch versions in other ways: suppose you were using the Mac App Store version of an app, and you switch to the non-Mac App Store one, which uses Sparkle for updates; using the Mac OS X user defaults system, the non-Mac App Store version will seamlessly pick up the user settings, while when Sparkle will ask for its settings (it has a few) in the application preferences it will find they are not set, and Sparkle will act as if there was no Sparkle activity previously for this application, in other words behave as expected. This is much more flexible than, for instance, a script to upgrade the preference file at each app update.

So, OK, the preference file need not contain every single setting the application relies upon before the application is run; what about after the application is run? By default, unfortunately, the Mac OS X user defaults system is not made aware of the decision taken by the application when the latter gets an empty value, so the user defaults system simply leaves the setting unset, and the preference file remains as-is. There is, however, a mechanism by which the application can declare to the user defaults system the value to use when the search for a given setting turns out empty, by registering them in the registration domain; but as it turns out, these default settings do not get written in the preferences file either, they simply get used as the default for each setting.

So in other words, when using techniques recommended by Apple, no setting will ever get written out to the preferences file until explicitly changed by the user. It is still possible to change the setting without using the application, but this requires knowing exactly the name of the setting, its location, and the kind of value it can take, which are hard to guess without it being present in the preference file. This will not do, so what can we do?

What I have done so far in my (not widely released) applications is the following (for my fellow Mac developers lucky enough to be maintaining Carbon applications, adapt using CFPreferences as appropriate): to begin with I always use -[NSUserDefaults objectForKey:] rather than the convenience methods like -[NSUserDefaults integerForKey:], at the very least as the first call, so that I can known whether the value is unset or actually set to 0 (which is impossible to tell with -[NSUserDefaults integerForKey:]); then if the setting was not found in the user defaults system, I explicitly write the default value to the user defaults system before returning it as the asked setting; for convenience I wrap the whole thing into a function, one for each setting, looking more or less like this:

NSString* StuffWeServe(void)
{
   NSString* result = nil;
   
   result = [[NSUserDefaults standardUserDefaults] objectForKey:@"KindOfObjectWeServe"];
   if (result != nil) // elided: checks that result actually is a string
      return result;
   
   // not found, insert the default
   result = @"burgers";
   [[NSUserDefaults standardUserDefaults] setObject:result forKey:@"KindOfObjectWeServe"];
   
   return result;
}

It is important to never directly call NSUserDefaults and always go through the function whenever the setting value is read (writing the setting can be done directly through NSUserDefaults). I only needed to do this for a handful of settings, if you have many settings it should be possible to implement this in a systemic fashion by subclassing NSUserDefaults and overriding objectForKey:, to avoid writing a bunch of similar functions.

Using this code, after a new setting is requested for the first time it is enough to synchronize the user defaults (which should happen automatically during the application run loop) for it to appear in the preference file and so more easily allow it to be discovered and changed by end users, or more typically, system administrators.

by Pierre Lebeaupin on December 31, 2012  •  Permalink
Posted in Mac Community, Programming

Posted by Pierre Lebeaupin on December 31, 2012

http://wanderingcoder.net/2012/12/31/discoverable-defaults/

Patents

Patents and their application to software have been in the news lately: Lodsys and other entities that seem to have been created from whole cloth for that sole purpose are suing various software companies for patent infringement, Android is attacked (directly or indirectly) by historical operating system actors Apple, Microsoft and Oracle (as owner of Sun) for infringing their patents, web video is standardized but the codec is left unspecified as the W3C will only standardize on freely-licensable technologies while any remotely modern video compression technique is patented (even the ostensibly patent-free WebM codec is heading towards having a patent pool formed around it).

Many in the software industry consider it obvious that not only a reform is needed, but that software patents should be banned entirely given their unacceptable effects; however I haven’t seen much of a justification of why they should be banned, as often the article/blog post/editorial defending this position considers it obvious. Well, it is certainly obvious for the author as a practitioner of software, and obvious to me as the same, but it’s not to others, and I wouldn’t want engineers of other trades to see software developers as prima donnas who think they should be exempted from the obligations related to patents for no reason other than the fact it inconveniences them. So here I am going to expose why I consider that software patents actually discourage innovation, and in fact discourage any activity, in the software industry.

Why the current situation is untenable

Let’s start by the basics. A patent is an invention that an inventor, in exchange for registering it in a public office (which includes a fee), is given exclusive rights to. Of course, he can share that right by licensing the patent to others, or he can sell the patent altogether. Anyone else using the invention (and that includes an end user) is said to be infringing the patent and is in the wrong, even if he came up with it independently. That seems quite outlandish, but it’s a tradeoff that we as a society have made: we penalize parallel inventors who are of good faith in order to better protect the original inventor (e.g. to avoid copyists getting away with their copying by pretending they were unaware of the original invention). Of course, if the parallel inventor is not found to have been aware of the original patent, he is less penalized than if he were, but he is penalized nonetheless. The aim is to give practitioners in a given domain an incentive to keep abreast of the state of the art in various ways, including by reading the patents published by the patent office in their domain. In fields where the conditions are right, I hear it works pretty well.

And it is here we see the first issue with software patents: the notorious incompetence of the USPTO (United States Patent and Trademark Office)1, which has been very lax and inconsistent when it comes to software patents, and has granted a number of dubious ones; and I hear it’s not much better in other countries where software patents are granted (European countries thankfully do not grant patents on software, for the most part). One of the criteria when deciding whether an invention can be patented is whether it is obvious to a practitioner aware of the state of the art, and for reasonably competent software developers the patents at the center of some lawsuits are downright obvious inventions. The result is that staying current with the software patents that are granted is such a waste of time that it would sink most software companies faster than any patent suit.

Now, it is entirely possible that the USPTO is overworked with a flood of patent claims which they’re doing their best to evaluate given their means, and the bogus patents that end up being granted are rare exceptions. I personally believe the ones we’ve seen so far are but the tip of the iceberg (most are probably resting along with more valid patents in the patent portfolios of big companies), but even if we accept they are an exception, it doesn’t matter because of a compounding issue with software patents: litigation is very expensive. To be more specific, the U.S. patent litigation system seems calibrated for traditional brick and mortar companies that produce physical goods at the industrial scale; calibrated in the sense of how much scrutiny is given to the patents and the potential infringement, the number of technicalities that have to be dealt with before the court gets to the core of the matter, how long the various stages of litigation last, etc. Remember that in the meantime, the lawyers and patent attorneys gotta get paid. What are expensive but sustainable litigation expenses for these companies simply put most software companies, which operate at a smaller scale, out of business.

Worse yet, even getting to the point where the patent and the infringement are looked at seriously is too expensive for most companies. As a result, attackers only need to have the beginning of a case to start threatening software developers with a patent infringement lawsuit if they don’t take a license; it doesn’t matter if the attacker’s case is weak and likely to lose in court eventually, as these attackers know that the companies they’re threatening do not have the means to fight to get to that point. And there is no provision for the loser to have to pay for the legal fees of the winner. So the choice for these companies is either to back off and pay up, or spend at least an arm and a leg that they will never recover defending themselves. This is extortion, plain and simple.

So even if bogus patents are the exception, it is more than enough for a few of them to end up in the wild and used as bludgeons to waylay software companies pretty much at will, so the impact is disproportionate with the number of bogus patents. Especially when you consider the assailants cannot be targeted back since they do not produce products.

But at the very least, these issues appear to be fixable. The patent litigation system could be scaled back (possibly only for software patents), and, who knows, the USPTO could change and do a correct job of evaluating software patents, especially if there are disincentives in place (like a higher patent submission fee) to curb the number of submissions and allow the USPTO to do a better job. And one could even postulate a world where software developers “get with the program” and follow patent activity and avoid patented techniques (or license them, as appropriate) such that software development is no longer a minefield. But I am convinced this will not work, especially the latter, and that software (with possible exceptions) should not be patentable, for reasons I am going to expose.

Why software patents themselves are not sustainable

The first reason is that contrary to, say, mechanical engineers, or biologists, or even chip designers, the software development community is dispersed, heterogeneous, and loosely connected, if at all. An employee in IT writing network management scripts is a software practitioner; an iOS application developer is a software practitioner; a web front-end developer writing HTML and JavaScript is a software practitioner; a Java programmer writing line of business applications internal to the company is a software practitioner; an embedded programmer writing the control program for a washing machine is a software practitioner; a video game designer scripting a dialog tree is a software practitioner; a Linux kernel programmer is a software practitioner; an embedded programmer writing critical avionics software is a software practitioner; an expert writing weather simulation algorithms is a software practitioner; a security researcher writing cryptographic algorithms is a software practitioner. More significantly, every company past a certain size, regardless of its field, will employ software practitioners, if only in IT, and probably to write internal software related to its field. Software development is not limited to companies in one or a few fields, software practitioners are employed by companies from all industry and non-industry sectors. So I don’t see software developers ever getting into a coherent enough “community” for patents to work as intended.

The second reason, which compounds the first, is that software patents can not be reliably indexed, contrary to, say, chemical patents used in the pharmaceutical industry for instance2. If an engineer working in pharmacology wants to know whether the molecule he intends to work on is patented already, there are databases that, based on the formal description of the molecule, allow to find any and all patents covering that molecule, or allow the knowledge with a reasonably high degree of confidence that the molecule is not patented yet if the search turns up no result. No such thing exists (and likely no such thing can exist) for software patents, where there is at best keyword search; this is less accurate, but in particular cannot give confidence that an algorithm we want to clear is not patented, as a keyword search may miss patents that would apply. It appears that the only way to ensure a piece of software does not infringe patents is to read all software patents (every single one!) as they are issued to see if one of them wouldn’t cover the piece of software we want to clear; given that every company that produces software would need to do so, and remember the compounding factor that this includes every company past a certain size, this raises some scalability challenges, to put it lightly.

This is itself compounded by the fact you do not need a lot of resources available, or to spend a lot of resources or time, to develop and validate a software invention. To figure out whether a drug is worth patenting (to say nothing of producing it in the first place), you need a lab, in which you run experiments taking time and money to pay for the biological materials, the qualified technicians tending to the experiments, etc. Which may not work, in which case you have to start over; one success has to bear the cost of likely a magnitude more failures. To figure out whether a mechanical invention is worth patenting, you need to build it, spend a lot of materials (ones constitutive of the machine because it broke catastrophically, or ones the machine is supposed to process like wood or plastic granules) iterating on the invention until it runs, and even then it may not pan out in the end. But validating a software invention only requires running it on a computer that can be had for $500, eating a handful of kilojoules (kilojoules! Not kWhs, kilojoules, or said another way, kilowatt-seconds) of electrical power, and no worker time at all except waiting for the outcome, since everything in running software is automated. With current hardware and compilers, the outcome of whether a software invention works or not can be had in mere seconds, so there is little cost to failure of an invention. As a result, developing a software invention comparable in complexity to an invention described in a non-software patent has a much, much lower barrier of entry and requires multiple orders of magnitude less resources; everyone can be a software inventor. Now there is still of course the patent filing fee, but still in software you’ve got inventions that are easier to come up with, as a result many more of them will be filed, while they impact many more companies… Hmm…

Of course, don’t get me wrong, I do not mean here that software development is easy or cheap, because software development is about creating products, not inventions per se; developing a product involves a lot more (like user interface design, getting code by different people to run together, figuring out how the product should behave and what users want in the product, and many other things, to say nothing of non-programming work like art assets, etc.) than simply the inventions contained inside, and developing that takes a lot of time and resources.

Now let us add the possibility of a company getting a software patent so universal and unavoidable that the company is thus granted a monopoly on a whole class of software. This has historically happened in other domains, perhaps most famously with Xerox who for long held a monopoly on copying machines, by having the patent on the only viable technique for doing so at the time. But granting Xerox a monopoly on the only viable copying technique did not impact other markets, as this invention was unavoidable for making copying machines and… well, maybe integrated fax/printer/copying machine gizmos which are not much more than the sum of their parts, but that was it. On the other hand, a software invention is always a building block for more complex software, so an algorithmic patent could have an unpredictable reach. Let us take the hash table, for instance. It is a container that allows to quickly (in a sense formally defined) determine whether it already contains an object with a given name, and where, while still allowing to quickly add a new object; something computer memories by themselves are not able to do. Its performance advantages do not merely make programs that use it faster, they allow many programs, which otherwise would be unfeasibly slow, to exist. The hash table enables a staggering amount of software; for instance using a hash table you can figure out in a reasonable time from survey results the list of different answers given in a free-form field of that survey, and for each such answer the average age of respondents who picked that answer (as an example). Most often the various hash tables uses are even further removed from user functionality, but are no less useful, each one providing its services to another software component which itself provides services to another, etc. in order to provide the desired user functionality. Thanks to the universal and infinitely composable nature of software there is no telling where else, in the immensity of software, a software invention could be useful.

Back when it was invented, the hash table was hardly obvious, had it been patented everyone would have had to find alternative ways to accomplish more or less the same purpose (given the universal usefulness it has), such as trees, but those would themselves have become patented until they was no solution left, as there are only so many ways to accomplish that goal (given that in software development you cannot endlessly vary materials, chemical formulas, or environmental conditions); at that point software development would have become frozen in an oligopoly of patent-having companies, which would have taken advantage of being the only ones able to develop software to file more patents to indefinitely maintain that advantage.

Even today, software development is still very young compared to other engineering fields, even to what they were around the start of the nineteenth century when patent systems were introduced. And its fundamentals, such as the hardware it runs on and its capabilities, change all the time, such that there is always a need to reinvent some of its building blocks; therefore patenting techniques currently being developed risks having enormous impact on future software.

But what if algorithmic inventions that are not complex by software standards were not allowed patent protection, and only complex (by software standards) algorithms were, to compensate for the relative cheapness of developing an algorithmic invention of complexity comparable to a non-algorithmic invention, and avoid the issue of simple inventions with too important a reach? The issue is, with rare exceptions complex software does not constitute an invention bigger than the sum of individual inventions. Indeed, complex software is developed to solve user needs, which are not one big technical problem, but rather a collection of technical problems the software needs to solve, such that the complex software is more than the sum of its parts only to the extent these parts work together to solve a more broadly defined, non-technical problem (that is, the user needs). However this complex software is not a more complex invention solving a new technical problem its individual inventions do not already solve, so patenting this complex software would be pointless.

Exceptions (if they are possible)

This does leave open the possibility of some algorithmic techniques for which I would support making an exception and allowing them patent protection while denying it to algorithms in general, contingent on a caveat I will get into afterwards.

First of these are audio and video compression techniques: while they come down to algorithms in the end, they operate on real world data (music, live action footage, voice, etc.) and have shown to be efficient at compressing this real-world data, so they have more than just mathematical properties. But more importantly, these techniques compress data by discarding information that will end up not being noticed as missing by the consumer of the media once uncompressed, and this has to be determined by experimentation, trial and error, large user trials, etc. that take resources comparable to a non-algorithmic invention. As a result, the economics of developing these techniques is not at all similar to software, and application of these techniques is bounded to some, and not all, software applications, so it is worth considering keeping patent protection for these techniques.

Other techniques which are worth, in my opinion, patenting even though they are mostly implemented in software are some encryption/security systems. I am not necessarily talking here of encryption building blocks like AES or SHA, but rather of setups such as PGP. Indeed these setups have provable properties as a whole, so they are more than just the sum of their parts; furthermore, as with all security software the validation that such techniques work can not be done by merely running the code3, but only by proving (a non-trivial job) that they are secure, again bringing the economics more in line with those of non-algorithm patents, therefore having these techniques in the patent system should be beneficial.

So it could be worthwhile to try and carve an exception and allow patents for these techniques and others sharing the same patent-system-friendly characteristics, but if attempted extreme care will have to be taken when specifying such an exception. Indeed, even in the U.S.A. algorithm patents are formally banned, but accumulated litigations ended up with court decisions that progressively eroded this ban, first allowing algorithms on condition they were intimately connected to some physical process, then easing more and more that qualification until it became meaningless; software patents must still pretend being about something other than software or algorithms, typically being titled some variation of “method and apparatus”, but in practice the ban on algorithm patents is well and truly gone, having been loopholed to death. So it is a safe bet any granted exception, on an otherwise general ban on software patents should it happen in the future, will be subject to numerous attempts to exploit it for loopholes to allow software in general to be patented again, especially given the important pressure from big software companies to keep software patents valid.

So if there is any doubt as to the validity and solidity of a proposed exception to a general ban on software patents, then it is better to avoid general software patents coming back through a back door, and therefore better to forego the exception. Sometimes we can’t have nice things.

Other Proposals

Nilay Patel argues that software patents should be allowed, officially even. He mentions mechanical systems and a tap patent in particular, arguing that since the system can be entirely modeled using physical equations, fluid mechanics in particular, the entire invention comes down to math in the end like for software, so why should software patents be treated differently and banned? But the key difference here, to take again the example of the tap patent he mention, is that the part of math which is an external constraint, the fluid mechanics, are an immutable constant of nature. On the other hand with algorithm patents all algorithms involved are the work of man; even if there are external constraining algorithms in a given patent, due to legacy constraints for instance, these were the work of man too. In fact, granting a patent because an invention is remarkable due to the legacy constraints it has to work with and how it solves them would indirectly encourage the development and diffusion of such constraining legacy! We certainly don’t want the patent system encouraging that.

The EFF proposes, among other things, allowing independent invention as a valid defense again software patent infringement liabilities. If this is allowed, we might as well save costs and abolish software patents in the first place: a patent system relies on independent infringement being an infringement nonetheless in order to avoid abuses rendering the whole system meaningless, and I do not see software being any different in that regard.

I cannot remember where, but I heard the idea, especially with regard to media compression patents, of allowing software implementations to use patented algorithm inventions without infringing, so that software publishers would not have to get a license, while hardware implementations would require getting a license. But an issue is that “hardware” implementations are sometimes in fact DSPs which run code actually implementing the codec algorithms, so with this scheme the invention could be argued to be implemented in software; therefore OEMs would just have to switch to such a scheme if they weren’t already, qualify the implementation as software, and not have to pay for any license, so it would be equivalent to abolishing algorithm patents entirely.

  1. I do not comment on the internal affairs of foreign countries in this blog, but I have to make an exception in the case of the software patent situation in the U.S.A., which is so appalling that it ought to be considered a trade impediment.↩

  2. I learned that indexability was a very useful property that, in contrast to software patents, some patent domains did have, and the specific example of the pharmaceutical industry as such a domain, from an article on the web which I unfortunately cannot find at the moment; a search on the web did not allow me to find it but turned up other references for this fact.↩

  3. It’s like a lock: you do not determine that a lock you built is fit for fulfilling its purpose by checking that it closes and that using the key opens it; you determine it by making sure there is no other way to open it.↩

by Pierre Lebeaupin on December 7, 2012  •  Permalink
Posted in Patents, Programming, Technology

Posted by Pierre Lebeaupin on December 7, 2012

http://wanderingcoder.net/2012/12/07/patents/

PSA: Do not release ARMv7s code until you have tested it

If you are using a third-party SDK in your iOS app, you may encounter a problem when linking with the current Xcode release: in that case the linker errors with the following line in the build log:

ld: file is universal (2 slices) but does not contain a(n) armv7s slice: libexample.a for architecture armv7s

(with some libraries, the linker will instead output the following when it errors, but it’s the same general problem:)

ld: warning: ignoring file libexample.a, file was built for archive which is not the architecture being linked (armv7s): libexample.a
Undefined symbols for architecture armv7s:

One solution for this issue is to get an updated version of the SDK that has a library with an ARMv7s slice (provided such an update exists, of course, otherwise you have no choice but to apply the second solution). However you should do so only if you have an iPhone 5 to test on; otherwise, I strongly recommend you apply the second solution: go to your project settings, (or target settings, if they are overridden at the target level), and edit the Architectures setting from “armv7 armv7s” to just “armv7”.

Why? Well, only the iPhone 5 can run the variant of your app code (called a slice) compiled for ARMv7s, so if you build and eventually release an update to your app that includes ARMv7s support, you would be releasing code you have not tested yourself, which is a big no-no. Don’t do it. In fact even if you can test on an iPhone 5, there is likely no need to rush and add ARMv7s support in your app as the benefits are incremental at best, as far as I can tell (but do not take my word for it, measure!); I really can’t understand why Apple added ARMv7s support in such a way that existing projects start using it right away by default.

Look forward to a post detailing the benefits (if any) to adding an ARMv7s slice to your app, as well as updates to my existing posts, in the coming weeks.

This post was initially published with somewhat different contents as at the time the iPhone 5 was not actually available. In the interest of historical preservation, the original content has been moved here.

by Pierre Lebeaupin on September 16, 2012  •  Permalink
Posted in Programming

Posted by Pierre Lebeaupin on September 16, 2012

http://wanderingcoder.net/2012/09/16/no-armv7s-til-tested/

Developer command-line tools setup

After my previous post, Gregory Pakosz wondered why I was using xcrun, as he did not need to (turns out he set to install command-line developer tools in the Xcode prefs, so otool and friends were in /usr/bin). That got me thinking a bit about the setup for accessing command-line tools we can assume another developer has.

Starting from the olden days of Mac OS X and up until recently, the Developer Tools were a system-spanning install, most definitely not self-contained. In particular, either as standard or as an option offered at install, you could install command-line tools like the compiler and more in /usr/bin, directly accessible to your shell; and even if some disabled the option, they would add /Developer/usr/bin to their $PATH. As a result, you could just assume otool, libtool, gcc, etc. would be directly accessible in the environment of a fellow developer, and just give command lines directly using them when conversing with them by email, Twitter, blog posts, etc.

Then the iPhone SDK happened, and our numbers grew immensely (waving at you guys!), but habits were generally kept. And then Xcode 4 happened and emphasized being a one stop shop for your entire development process. And with Xcode 4.3 Xcode truly became self-contained, with no longer any installation per se; these days it is possible, and I suspect common, to develop and submit iPhone apps without needing to be aware of the Terminal at all (not that this is a bad thing, mind you).

In my case I initially missed installing the command-line tools as instead of meeting the customization step of the install process and enabling everything except WebObjects, now I simply had Xcode as an application in /Applications and never thought to seek the installs in the Xcode preferences.

I have now enabled that preference and installed the tools in /usr/bin, but still I missed it initially, so who knows how many others do. Also, it occurs to me more and more build systems and other scripts in the Mac/iOS development world are now aware of the Xcode hierarchy, and rely on xcode-select -print-path and/or an environment variable to locate the developer directory and use the tools in there directly; as a result, the Command Line Tools install is now mostly for the benefit of open source/Unix stuff which simply assumes cc/gcc is in the path, and maybe we should start thinking about it as being for that specific purpose.

So my question is, should we simply assume command-line developer tools are in the path and write our blogs posts, emails, and Twitter messages as usual (maybe with a reminder to “make sure you have the Command Line Tools installed in the Xcode prefs.”)? Or should we start considering these tools do not necessarily belong in /usr/bin and instead instruct our readers to use xcode-select -switch then xcrun, or even instruct them to directly add the Developer/usr/bin folder inside Xcode to the $PATH (which is likely to break from time to time as the paths change)? What do you think? As usual, write me at wanderingcoder@sfr.fr.

Okay, feedback is not unanimous in either direction. For now I will keep putting xcrun, with the assumption that it will help those who have not installed the Command Line Tools, while those who have will know enough to remove it from the command line before use. — August 8, 2012
by Pierre Lebeaupin on July 5, 2012  •  Permalink
Posted in Programming

Posted by Pierre Lebeaupin on July 5, 2012

http://wanderingcoder.net/2012/07/05/developer-cmd-line-tools/

Dealing with multiply defined symbols

ld: duplicate symbol _DoStuff in OneFile.o and CompletelyDifferentFile.o for architecture armv7

☠!#✺⁂☆⚡❕☭✹✊✨#❗

Like me, this is probably your spontaneous reaction upon getting this delightful error message when trying to build your app. This is a signal that at the very least some more work will be needed on the code you just integrated before your app will work with it. But it could be even worse than you think, as the Mac OS X linker (as of Xcode 4.3.2) will only report duplicate symbols one. at. a. time. So there could be in fact 50 colliding symbols, and the linker will only tell you about the next one only once you’ve fixed the previous one, making you require 50 prefix-compile-link cycles! Today I will show you how to efficiently assess the damage first thing, then show you different methods to fix the issue, depending on the situation.

The Problem

The issue here, at its most fundamental level, is that there exists two Objective-C classes with the same name, C functions with the same name, or C++ functions with the same name and signature (or possibly two global variables with the same name—it can happen) such that the linker cannot resolve references to this symbol, as it does not know which to pick. One could think the linker could pick either definition, but this would be an incredibly dangerous thing to do as some references likely expect the other definition of the code, and so would end calling completely (or subtly) different code than what it expected and you would end up with a mysterious, impossible-to-debug issue at runtime (and that’s if you’re lucky).

This kind of problem typically occurs at a time in your project which may not be the best: at the time of integration of a separate body of code (for instance, a third-party library); it may occur for various reasons, the most common one is that the code you just integrated contains utility functions it uses, but your code already contains utility functions of the same name because these utility functions were copy pasted one way or the other, and then likely modified. It is also possible to encounter external code which contains unprefixed functions or classes (always put your two or three letter organization prefix in front of any class or function which has visibility outside the current source file, people!), and one of those collides with one of yours, or one in another library with unprefixed symbols. Sometimes a whole module may be included in two different libraries you are using, and you mistakenly built both libraries with this module included, forgetting you were going to use them together. And in some cases the code you just added may include internally its own version of an open-source library like SQLite, which will collide with the SQLite system framework if you are using it.

Depending on the reason and the circumstances, you may have different constraints, so it is important to recognize which situation you are in in order to apply the most appropriate solution; you don’t want to prefix 100 functions on a deadline when there is a better way.

Assessing the damage

The first order of business is to figure out how many colliding symbols there are. But how to do it if the linker is going to only report one such symbol before giving up? I don’t have it down to a single script that would do all the steps, but here is the process I followed when I found myself in this situation:

Disclaimer: these instructions and Terminal commands come with no warranty, I cannot be held responsible if you hose your computer following them; caveat emptor.

  1. I arranged to be able to use commands that come as part of the Xcode tools package; I did so with xcode-select -switch /Applications/Xcode.app/Contents/Developer, and preceding all developer tool commands by xcrun; I could also have added the tools folder to my path.

  2. In one case I came across, the colliding symbols were already in the same static library before the final link (it was an intermediate static library generated by a subtarget) and I could start with step 3.

  3. I generated a static library with the object files that would be linked, by taking from the linker invocation that failed (in the Xcode build log) the -filelist option with its parameter (a file ending in .LinkFileList), as well as any static library parameter (the files ending in .a), and putting them right after “xcrun libtool -arch_only armv7 -o stuff.a ” (with a space after stuff.a) in a Terminal window, to generate a library named stuff.a

  4. Then I ran this wonderful command (as one line):

    xcrun otool -vS stuff.a | LANG=C sed -n "/^object *symbol name\$/,\$p" | LANG=C sed "1d" | LANG=C sed "s/^.* //g" | LANG=C sort | LANG=C uniq -d > duplist

    (it may be useful to understand what this does: otool -vS reads the table of contents of the library, which lists the symbols and the object file where the symbol can be found; the first two sed commands extract the relevant part from the output; the third removes the file name part of each line; sort, then uniq -d extract the lines which appear more than once)
    At this point, duplist contained the list of duplicate symbols, one per line.

  5. I ran this command next:

    xcrun otool -vS stuff.a | LANG=C grep -f duplist > dupreport

    At this point, dupreport listed the object files containing one of the duplicate symbols, as well as which of the duplicate symbols they contain.

Now I had a count of the duplicate symbols (there were only a handful, if it wasn’t the case I would have used wc -l duplist to count them), all such duplicate symbols, and which object files they occur in.

Notice this won’t work to list duplicate symbols with frameworks; you will want to handle that case specifically, anyway.

Fixing the damage

We are getting there: now we know the situation, we can fix it in one fell swoop. But we must avoid making non-trivial changes to the code in doing so, or we risk introducing bugs, while we were merely supposed to integrate already working code together.

If there are only a handful of duplicate symbols, simply prefix the functions and all the places they are called with a different prefix depending on the side; e.g. instead of PREPackBits, you would have PREReaderPackBits and PREWriterPackBits. Unless you only have the code for one side, it’s preferable to prefix both sides (that way any reference that you forgot to prefix will cause an error when linking, rather than silently resolving to the wrong version); note that even if you control both sides, it is not a good idea to try and merge the function implementations so that both sides call a single function which would satisfy them both: even if the function was copy-pasted from one side to another, it was likely modified, and at this point you are trying to integrate two bodies of code that work well separately, it is not a good time to make semantic changes to the code. If the code duplication bothers you, it will always be possible to refactor later. If you have the code for neither side, then you are in trouble, though you may be able to apply the partial link technique described later.

If the duplicate symbols are all the functions in a submodule used by both sides, then you made a mistake when building the libraries and included that submodule in both libraries; since on both sides the same source files with the same compile options are presumably used, then you should build and include that submodule in one library only, and leave it out of the second one: code in the second library which relies on the submodule API will simply use the one from the first library.

If the symbol collision involves a system framework and appears to be a limited coincidence (e.g. an unprefixed function name which is also used in a system framework), then just prefix the function on your side with your organization prefix. If, however, the collision is not a coincidence, for instance because you embed in your app an open-source project which is also available as a system framework, such as SQLite, then it is likely not practical to prefix all the functions in your internal copy of the open-source project, and neither it is to use the system framework and remove your internal version of the open-source code (which may be a different version, have some customizations, and what not). What I did in such a case, at least as a first step, was to do a partial link on the component of my app which made use of SQLite. A partial link forces references to be resolved, but produces an object file that can then be subject to further linking; this is done with ld -r -ObjC, and the linker invocation has to be given (with the -exported_symbols_list option) a file containing the function names one would like the generated object to export, this would be the API of the component. That way, references to SQLite functions coming from inside the component get resolved correctly, and from then on these functions are no longer visible outside the component, such that the final link can proceed without problem.

This is a bit of an extreme solution, as it forces you to handle that component specially (there is no support for this in Xcode), and you will have to redo this handling every time you need to rebuild the component, but it got me out of a bind without having to make changes to my code. In the long run however, you should either prefix or get rid if that internal version of the open source project.

Finishing your work

Now complete the integration of the body of code you were intending to add (fixing runtime issues, etc.) Once your app is running satisfactorily again, then it is probably time to apply a definitive solution to the original duplicate symbol issue: for instance, you may find it unsustainable to have parts of your code use a modified by mostly duplicate function and would much prefer there to be a single, unified copy. Now is a good time to do it, or maybe later; what matters is that this refactoring not be done before you get the app running again, as then you would have no idea whether bugs were due to the newly integrated code, or due to the changes you made in order to integrate it. The sum it up: get the newly integrated code running while making as few changes as possible first, then only refactor the code to your liking.

by Pierre Lebeaupin on June 30, 2012  •  Permalink
Posted in Programming

Posted by Pierre Lebeaupin on June 30, 2012

http://wanderingcoder.net/2012/06/30/multiply-defined-symbols/

Beware of ARMv6-only iOS libraries

“What the f…1 is going on?”

There I was, one day at work, trying to figure out why this iOS project wouldn’t link, giving the error “Undefined symbols for architecture armv7:” with the missing symbols being the entry points of a third-party library that was just added. Which made no sense at all, as the library did define these symbols (I checked using otool -vS), and the library was properly pulled in by a subproject, which was itself properly pulled in by the app (I checked with the log of commands executed by Xcode). I already tried cleaning all projects three times, nuking all the caches I could think of, and removing and reinserting the external library, to no avail. I was at a loss for ideas.

This is an actual issue that I want to warn you about because it could affect any iOS developer, so I’ll cut to the chase: the issue was that the third-party library was only built for ARMv6 (or only built for for the ARM_ALL subtype, the same issue occurs in the end) while I was trying to build an app with an ARMv7 slice.

This can happen with libraries that haven’t been updated recently, or that work fine in ARMv6 for most people (I’ll explain why in a minute) so are not built for both ARMv6 and ARMv7. If this happens to you (you can check with otool -vhf <library> | less; notice that in this context, “ALL” does NOT mean “both ARMv6 and ARMv7”), demand from the supplier of the library that you be provided a library that has both ARMv6 and ARMv7 slices; and if you have no choice but to take matters into your own hands (as I had to do), it’s possible to build a simple tool (everything you need is in mach-o/loader.h and mach/machine.h) that will manipulate the ARMv6 object files so that they become ARMv7 (the code in fact remains unchanged) so that you can craft a suitable library with both ARMv6 and ARMv7 slices.

What’s interesting is that the issue only becomes apparent in a specific configuration: if the external library is referenced not by the app target, but by a library target which is itself used by the app. iOS apps projects often have only one target, as iOS apps are generally simpler than Mac apps, but it’s not outlandish for more complex iOS projects to have multiple targets which depend on each other, in which case you could encounter the issue.

What is happening here is that ARMv6 and ARMv7 are treated separately enough by the iOS toolchain that they are in practice treated as different architectures altogether, a bit like x86 and ppc are when building Universal Binaries (and this is no fluke, there are plenty of ARM-specific patches in the source to enforce this behavior). What this means among other things is that libtool (the tool which builds libraries) will refuse to mix together ARMv6 and ARMv7 object files; instead, it will combine them separately, then put the results side by side in a fat library.

So in our case what happens is that the object files from the third-party library, being ARMv6, are put together with the ARMv6 object files (of which there may be none, if you’re building ARMv7-only) in the ARMv6 slice of the library target, while the ARMv7 object files are put together in the ARMv7 slice, which therefore does not include anything from the third-party library. Then when the app target is built, more precisely when its ARMv7 slice is built, the linker will find the ARMv7 slice of the library from the target, and will not look in the ARMv6 slice (as it is expected to define the same symbols, so the linker would find duplicate definitions if it tried); so the third-party library object files are never found, referenced symbols are undefined, and you wonder what is going on.

This does not happen if the third-party library is referenced directly by the app target, as then the linker will find this library with only an ARMv6 slice, in which case there is an exception to the segregation rules: since there is no ARMv7 version, these ARMv6 object files will be searched for symbols and linked in to satisfy the dependencies instead, even though the linker is building for ARMv7; the linker is the only one allowed to mix together ARMv6 and ARMv7 object files. So the ARMv6-only library is working in this case, which may lead the supplier of the library to believe the library does not need to be updated to have an ARMv7 slice, while in fact it does need to, otherwise it will cause problems for people who have more complex project structures.

  1. do not think I am too prude to be spelling out the f-word here; I am just saving f-bombs on this blog for when they are really worth it↩

by Pierre Lebeaupin on May 11, 2012  •  Permalink
Posted in Programming

Posted by Pierre Lebeaupin on May 11, 2012

http://wanderingcoder.net/2012/05/11/armv6-ios-libraries/

A few things I would have liked to read about in John Siracusa’s Lion review

Yesterday we saw a few things that John Siracusa didn’t mention in his Snow Leopard review but that I think could have been in talked about in there. Today we will do the same with his Lion review.

The same disclaimer applies: John can’t know everything or mention everything, so do not construe anything I say here as being any sort of criticism of his work, much to the contrary in fact.

First, since Lion requires a 64-bit Mac, I was wondering whether built-in executables actually were 64-bit only, as it would cut short any attempt to run this release of Mac OS X on anything older than its baseline requirements (some earlier releases of Mac OS X could be massaged to do so, to an extent). As it turns out, most executables do have a 32-bit slice, with the exception however of the Finder, which therefore prevents Lion from meaningfully running on a 32-bit machine.

Then, the specific system requirements for AirDrop made me wonder what was the deal here, and whether there was any relationship with Wi-Fi Direct. Unfortunately, I do not know much here (besides that there is indeed a relation), so John covering this would have been all the more welcome. Ah well.

The addition of the AVFoundation APIs (originally released on iOS) to Mac OS X raises an important question that I haven’t seen addressed: what about QTKit? Does it turn out not to be the way forward (even though QTKit is pretty recent), or are they meant to integrate, or is QTKit more for a certain class of needs and AVFoundation for others, or something else altogether? I have no idea, and that’s one of the of the first things (admittedly, as an iOS multimedia app developer, I am a bit biased) I wanted to know. Maybe in the next review John will make a return to covering video technologies at length…

Lastly, I’ve been very intrigued by the rise of SSDs (though I don’t use one myself) and the impact on how storage devices are abstracted, and so a word about Lion support for TRIM would have been nice. Apparently Apple is quietly and selectively enabling it for some drives, but this comes from sources which aren’t very authoritative, while on the other hand with John this would have been coming, not from the horse’s mouth, but pretty much the next best thing.

by Pierre Lebeaupin on April 2, 2012  •  Permalink
Posted in Mac Community, Programming, Technology

Posted by Pierre Lebeaupin on April 2, 2012

http://wanderingcoder.net/2012/04/02/complements-10-7-review/

A few complements to John Siracusa’s Snow Leopard review

I always read John Siracusa’s review of the latest Mac OS X release with great interest, and he always delivers. Such is the case again with his review of Mac OS X Snow Leopard, which is of particular interest as for this release Apple teased only “one new feature” (Microsoft Exchange support), and John does a bang up job showing the other, non-obvious new stuff in this release (as obviously the guys writing Mac OS X did not just stop working for nearly two years).

That being said, there are a few things that I either heard about before, or learned about afterwards, that I wish he would have covered in the review. This isn’t necessarily a reproof of his review, as he has to make choices as to what he covers, and he can’t be aware of everything anyway. Nevertheless, here are a few things I wish I could have read about in his review.

QuickTime X

John does of course cover QuickTime X, but there is some more interesting stuff I found out about it.

First, the offloading mechanism he describes, where videos that QuickTime X cannot handle are played in a QTKitServer process, is not sufficient for all cases: there are videos that the new QuickTime Player app cannot even handle and have to be opened by QuickTime Player 7. This is why, when you install QuickTime Player 7 (which is an optional install), video files become owned by a “QuickTime Player Launcher” app (which lives inside QuickTime Player 7), and whenever you open a video this app dispatches the video to the new QuickTime Player, or to QuickTime Player 7 if the former cannot handle it. It’s safe to say that Apple has you covered for this transition.

Second, another feature not supported by QuickTime X at this time is video orientation, which is surprising since it is used by the video recording feature of the iPhone 3GS; as a result, if you have videos recorded in tallscreen from your iPhone, the new QuickTime Player will play them rotated 90°.

Third, contrary to what he implies in page 5: “the most significant new 64-bit-only API is QuickTime X”, 32-bit apps can use the QuickTime X engine; indeed I know of a bug in the traditional QuickTime playback engine, which I can reproduce easily in QuickTime Player 7, but cannot with the new QuickTime Player, whether it is running in 32 or 64-bit mode.

libcache

libcache is an interesting new functionality from the lower levels of Mac OS X. Back in the days of classic MacOS, memory had to be much more explicitly managed, and one of the ways it was done was with purgeable memory buffers which could be freed by the operating system if it needed the memory; a common pattern was loading data from a file in such a buffer, if when coming back to it the application found the data still there, all the better, otherwise it would simply allocate the buffer again and reload the data from disk. The Mac OS X virtual memory system and memory-mapped files advantageously replace this pattern and purgeable buffers are never purged on Mac OS X. However, a use case for which there was no longer a solution is the following: what about data which is computed from other data and cheaper to recompute than to save then reload from disk? An application willing to cache this data either doesn’t, and wastes performance recomputing it each time, or keeps it in memory, and if this data gets paged out to disk when the application then tries to use it again the virtual memory system wastes time reloading it from the pagefile as the application could have recomputed it more quickly instead.

Enter libcache. With libcache programmers can attach data to a cache object and libcache will call a release callback on the data object if memory needs require it. This functionality is also available at the Foundation level, and Foundation also provides an equivalent of the purgeable buffers of old with NSPurgeableData.

This is not too frequent but still a pattern I see with Mac OS X: it generally introduced significantly easier ways of doing common operations, replacing mechanisms from classic MacOS which needed much more manual management, but some special use cases ended up not being well-served by the new Mac OS X way, and Mac OS X ends up rediscovering and supporting these some time later. Grand Central Dispatch, interestingly, is the exact converse: a massively simpler way to leverage functionality (here, multicore processing) that will ensure this functionality will end up being actually used in much more software, improving performance and user experience across the board.

Tomorrow, we’ll see a few more things I wanted to read about in John Siracusa’s Lion review.

by Pierre Lebeaupin on April 1, 2012  •  Permalink
Posted in Mac Community, Programming, Technology

Posted by Pierre Lebeaupin on April 1, 2012

http://wanderingcoder.net/2012/04/01/complements-10-6-review/

Developer ID might not seem restrictive, but it is

I need to talk about Gatekeeper and Developer ID.

In short, I am very uncomfortable with this previewed security feature of Mountain Lion. Apple is trying to assure that users are only going to be safer and that developers are still going to be able to do business as usual, but the Mac ecosystem is not limited to these two parties and this ignores pretty much everyone else: for these people Gatekeeper is going to be a problem. Enough so to make me consider switching.

I don’t mean to say it’s all bad, as Apple is set to allow more at the same time as it allows less. Indeed, with Developer ID Apple is clearly undertaking better support of apps from outside the Mac App Store, if only because they will have to maintain this system going forward, and I can only hope this support will improve in other areas (such as distribution: disk images are long past cutting edge). But while Apple gives with one hand, it takes away with the other, as Mountain Lion will by default (at least as of the current betas, though that seems unlikely to change) reject unsigned apps and apps signed by certificates other than Mac App Store and Developer ID ones; of course most people will not change that default, and so you will have trouble getting these people to run your code unless you get at least a Developer ID from Apple, and while better than requiring you to go through the Mac App Store this requirement is quite restrictive too.

The matter is not that with Developer ID apps will now be exposed to being blacklisted by Apple; honestly, speaking as a developer I personally do not mind this bit of accountability. Maybe there are going to be issues with this power now entrusted to the hands of Apple, such as the possibility of authorities (through executive branch bullying, or with a proper court order) asking Apple to neutralize an app perceived as illegal, but if this ever happens I believe the first incidents will cause this eventuality to be properly restricted by law.

No, the matter, as I wrote to Wil Shipley in an email after his proposal, is that many people who are important to the Mac platform are going to be inconvenienced with this, as getting a Developer ID requires a Mac Developer Program membership.

  • Sysadmins/IT people, to begin with, often need to deploy scripts, and either those don’t need to be signed, and they become the new malware vectors, or they do (Apple could define an xattr that would store the signature for a script) and then any company deploying Macs needs to enter the Mac Developer Program and manage a Developer ID that IT needs to be able to access day to day (that is, not just for releases, like in a software company) and so could leak, just so that the company can service its own Macs internally.

  • Then we have people using open-source tools that Apple doesn’t provide, such as lynx, ffmpeg, httrack, mercurial, etc., and who likely get them from projects like MacPorts; maybe we have an exception for executables that were built on the same machine, but how is it enforced then?

  • Student developers have historically been very important to the Mac platform, if only because many current Mac developers started out as such. If entering the Mac Developer Program is required to distribute Mac apps in the future, it’s a threshold that many will not clear, and as a result they will not get precious feedback from other people using their code, or worse they will not choose Mac development as a career as they could have if they had been encouraged to do so by people using their software (for instance, Jeff Vogel wasn’t planning on making Mac games as a career, but he quit grad school when Exile started becoming popular). At 99$ (per year), it seems silly to consider the cost of the Mac Developer Program as an obstacle, especially when compared to the cost of a Mac, but you have to consider the Mac likely benefitted from a student discount and was possibly entirely paid by the family; not so for the Mac Developer Program. Regardless, any extra expense will, rationally or not, cause it not to be taken by a significant portion of the people who would have otherwise tried it, even if it would have paid for itself eventually.

  • Many users will tinker with their apps for perfectly legitimate reasons, for instance in order to localize it and then submit the localization to the author, or in the case of games to create alternate scenarios or complete mods. It’s something that I am particularly sensitive to, as for a long time I have both enjoyed other’s people’s mods and conversely tinkered myself and shared with others: I have created mods, documented the formats to help others create mods, extracted data from the game files, gave tips and tricks and development feedback on other people’s in-progress mods, I was even at some point in charge of approving mods for a game to the official mods repository, and I created tools to help develop mods (more on that later). The user modding tradition is very strong in the Ambrosia Software games community, going back to Maelstrom nearly 20 years ago, and that’s merely the one community I am most familiar with. However, tinkering in such ways typically breaks the app signature; an app with an invalid signature will currently run on Lion (I know it if only because my Dock currently has an invalid signature), but it will likely change with Mountain Lion as otherwise Gatekeeper would be pointless (an important attack to protect against is legitimate apps that have been modified to insert a malicious payload and then redistributed). So we will have to rely on developers excluding files that could be desirable for users to tinker with from the signature seal… well, except that the developer will then need to make sure the app cannot be compromised if the files outside the seal are, and I’m pretty sure it’s impossible to do so for nibs for instance, so app developers will not be able to simply leave the nibs out of the seal so that users may localize them; they will need to roll out systems like the one Wil Shipley developed for localizations, completely out of the realm of Apple-provided tooling.

  • Power users/budding developers will often create small programs whose sole purpose is to help users of a main program (typically a game, but not always), for instance by interpreting some files and/or performing some useful calculations; they typically develop it for themselves, and share it for free with other users in the community of the main program. It’s something I have done myself, again for Ambrosia games, and it’s a very instructive experience: you start with an already determined problem, file format, etc., so you don’t have to invent everything from scratch which often intimidates budding developers. However, if it is required to register in the Mac Developer Program to distribute those then power users will keep those to themselves and they won’t benefit from the feedback, and other users won’t benefit from these tools.

(Note that Gatekeeper is currently tied to the quarantine system, and so in that configuration some of the problems I mentioned do not currently apply, but let’s be realistic: it won’t remain the case forever, if only so that Apple can have the possibility of neutralizing rogue apps even after they have been launched once.)

In fact, a common theme here is that of future developers. Focusing solely on users and app developers ignores the fact that Mac application developers don’t become so overnight, but instead typically start experimenting on their spare time, in an important intermediate step before going fully professional; it is possible to become an app developer without this step, but then the developer won’t have had the practice he could have gotten by experimenting before he goes pro. Or worse, he will have experimented on Windows, Linux, or the web, and gotten exactly the wrong lessons for making Mac applications—if he decides he wants to target the Mac at all in the end.

Because of my history, I care a lot about this matter, especially the last two examples I gave, and so I swore that if Apple were to require code to be signed by an authority that ultimately derives from Apple in order to run on the Mac, such that one would have to pay Apple for the privilege to distribute one’s own Mac software (as would be the case with Developer ID), then I would switch away from the Mac. But here Apple threw me a curveball, as it is the case by default, but users can choose to allow everything, but should that matter, since the default is what most people will ever know? Argh! I don’t know what to think.

In fact, at the same time I am worrying about the security of the whole system and wish for it to be properly secure: I know that any system that allows unsigned code to run is subject to the dancing bunnies problem; and maybe the two are in fact irreconcilable and it is reality itself I am having a problem with. I don’t know. Maybe Apple could allow some unsigned apps to run by default, on condition they have practically all the sandboxing restrictions to limit their impact. The only thing is, in order to be able to do anything interesting, these apps would at least have to have access to files given to them, and even that, combined with some social engineering, would be enough for malware to do harm, as users likely won’t treat these unsigned apps differently from regular desktop apps, which they consider “safe”. Maybe the only viable solution for distribution of tinkerer apps are as web apps (I hear there is work going on to allow those to access user files); I don’t like that very much (e.g. JavaScript is not very good to parse arbitrary files), but at the same time users do tend to take web apps with more caution than they take desktop apps (at least as far as giving them files goes, I hope), and any alternate “hyper sandboxed” system that would be introduced would have to compensate the 15+ years head start the web has in setting user expectations.

The same way, the very same cost of the Mac Developer Program which is a problematic threshold for many is also the speed bump that will make it economically unviable for a malware distributor who just had its certificate revoked to get a new one again and again.

This is why, paradoxically, I wish for iOS to take over the desktop, as by then iOS will likely have gained the possibility to run unsigned apps, and users having had their expectations set by years of being able to use only (relatively) safe iOS App Store apps will see these unsigned apps differently than they do apps from the store.

Anyway, nothing that has been presented about Mountain Lion so far is final, and important details could change before release, so it’s no use getting too worked up based on the information we know today. But I am worried, very worried.

by Pierre Lebeaupin on March 7, 2012  •  Permalink
Posted in Programming, Technology

Posted by Pierre Lebeaupin on March 7, 2012

http://wanderingcoder.net/2012/03/07/developer-id-restrictive/

« Older Posts