Application Cache was fired for his douchebaggery

To all of you who enquired about the whereabouts of Application Cache, I regret that I have to inform you that he is no longer with our company. This was not an easy decision to take, but we believe it was the right one.

While it has been no secret for some time that Application Cache was a douchebag, this was not necessarily apparent at first. Application Cache promised so much, and we believed him because he could prove his claims to a large extent. However, his way of working was so much at odds with the way other web components work (especially long-time pillar of web infrastructure HTTP cache) that his core value proposition was harder to exploit than it should have been (with many unfortunate pitfalls, as Jake Archibald documented); and worse, his more advanced promises, while working in basic scenarios, had some ancillary troubles, which unexpectedly turned out to be intractable no matter how hard we tried, and so these promises never came to light.

Because he was useful despite the issues, we tried to work with him on these, with many counseling sessions with HR; however, Application Cache was adamant that this was his fundamental mode of operation and he could not work any other way, and that others would have to adapt to him. This, of course, was not remotely acceptable, but we could not find any way to make him change either, so little progress was made. There was some, as we did manage to make him more transparent; some claimed that made him no longer a douchebag, but in truth he remained one.

Still, we believed that it could still be worth keeping him just for his core value proposition of using web apps while offline. But as time went on, it became clear that even that was not going to be worth the bother, again as a consequence of his fundamentally different way of working. Things came to a head when we tried to solve race conditions resulting from the possibility that a user load the initial HTML page before the web app is updated, and its dependencies (including the manifest) after the web app is updated: the manifest has to be updated at the same URL (it acts as a fixed entry point of sorts for users who already have the web app in Application cache), so we could not rely on the HTML pointing to a new manifest URL so that the update of the entry point would atomically result in the update of the web app. Even with the provision that the manifest be redownloaded after the entry point, and checked against the manifest downloaded before in the case of an app already in Application Cache (so as to try to have the manifest always loaded after the entry point, at least conceptually), we were stuck.

Some solutions were found, though limited to ideal situations; there was no solution available for the case of a serving infrastructure, such as content distribution networks, with only “eventually consistent” or other weak guarantees, and there was no solution either if even minimal use of FALLBACK: was required. Moreover, even in ideal situations those solutions bring a lot of burden on the web developer, too much burden considering that offline web apps ought to work correctly in the face of these race conditions by default, or at least with minimal care. In the end, Application Cache was let go a few months ago.

If you were relying on the services provided by Application Cache, don’t worry. While there will be no future evolution (in particular, don’t expect bugs to get fixed), a new guy was hired to perform the tasks of Application Cache exactly as the latter did them. This new guy, Service Worker, will also provide a new service allowing web apps to work offline, this time in harmony with the other web components: for instance, out of the box he makes it possible to throttle checks for updated versions simply by setting a cache control header on the service worker (the period being a day at most); something which was exceedingly hard, if not impossible, with Application Cache due to his bad interactions with HTTP cache. He was already available in Chrome, and with the recently released Firefox 44, two independent, non-experimental implementations have now shipped, so you should take the time to make his acquaintance.

New software release: JPS

JPS (stands for Javascript Patching System) is a web app that applies binary patches, currently IPS patches. Usage is simple enough: you provide the reference file and the patch file, and once patching is done you recover the patched file just as you would download a file from a server, except everything happens on your local machine. Moreover, JPS works while offline, thanks to Curtain, which was in fact developed for the needs of JPS.

JPS works on any reasonably recent version of Firefox or Chrome (both of which update automatically anyway), as well as any version of Opera starting with Opera 15. Unfortunately, some of the features used (download of locally-generated files in particular) are not universally supported yet, which means that, regardless of my efforts, Safari (rdar://problem/23550189, OpenRadar) and Internet Explorer are not supported; as a Safari user myself, this bothers me, but I could not find any way around this issue, you will have to wait for a version of Safari that supports the download attribute.

Some background…

My motivation for writing JPS came from two events:

Indeed, when I learned of Zelda Starring Zelda I wanted to play it (A+++ would play again, currently playing the second installment), but realized the IPS patcher I previously used no longer ran (it was built for PowerPC), and while I was able to download and use a different patcher I thought there had to be a better way than each platform using a different program, program also susceptible to becoming unsupported. And this joined my thoughts from the time when Gatekeeper and Developer ID were announced, where I wondered if we couldn’t circumvent this Apple restriction using web apps. So I decided I would develop a web app to apply IPS patches.

While most of the difficulties were encountered when developing the Curtain engine, the browser features used by JPS itself, namely client-side file manipulation and download, led to some challenges as well. One fun aspect was taking a format, IPS, which embeds many assumptions, some undocumented, on C-like manipulation APIs (e.g. writing to a mutable FILE*-like object, and performing automatic zero filling when writing past the end of file), and making it work using the functional Blob APIs, based on slicing and concatenation of arrays and immutable Blob objects. There were a few interesting surprises, for instance early versions of JPS could, on some input files, cause Firefox to crash, taking down JPS and all the other Firefox tabs! Worse, resolving this required a significant rewrite to the patching engine, which led me to develop automated tests to catch any regression before performing this rewrite, to ensure that the rewrite would not regress in any way (it didn’t).

JPS has been extensively tested prior to this release; I tested myself about a hundred patches, with only one patch not working while running on Firefox (bug report), and it has been in open beta for some time without any other problematic patch having been reported.

The JPS source code is available under a BSD license; the source release contains all the needed code to deploy it with Curtain (which has to be downloaded separately), as well as test vectors for the IPS file format and a test harness to automatically test JPS using these files.

A few more words

While I would have liked to support Safari so that JPS could run out of the box on Mac OS X, I deem this proof of concept of a desktop-like web app to be good enough for at least a subset of desktop use cases; enough so for me to put the Gatekeeper and Developer ID concerns behind me. I can now reveal that, because of these concerns, I did not update to Mac OS X Mountain Lion or any later version until today; yes, up until yesterday I was still running Lion on my main machine.

Now that JPS and Curtain have been released, I can’t wait to see what will be done with this easy (well, OK, easier) way to develop small desktop-like tinkerer tools using the web!

Introducing Curtain

I’m excited to introduce Curtain to you today. Curtain is a packaging and deployment engine for desktop-like web apps; Curtain handles the business of generating the app from source files and deploying it on the server such that it supports offline use.

Curtain can be cloned from BitBucket, and it has a sample app, both under the BSD license. Rather than repeating the Readme found there, I would like here to provide some background.

Some background…

Offline support

I wanted to use Application Cache for a project; as you know, Application Cache is a douchebag, but even that article did not prepare me for how much of a douchebag it is. In particular, you want web apps to be able to be updated, if only because the first version inevitably has bugs. Remember that, even if the list of files in the manifest does not change, the manifest has to change whenever the app changes, otherwise users won’t get the updated version. So how to update the manifest and app?

  • If the app is updated in this manner:

    1. manifest is updated
    2. remainder is updated

    or even if the two are updated at the same time, then you could run into the following scenario:

    1. user does not have the app in cache, and fetches the HTML resource
    2. manifest is updated
    3. remainder is updated
    4. due to a network hiccup on her side, user only now fetches the manifest

    Now the user has the manifest for the updated version, but is really running the previous version of the web app. Even if the list of cached files is still correct, now whenever the user agent checks for an updated manifest it will find it to be bit-for-bit identical, and the user agent will not update the version the user uses, which is out of date, until a second update occurs. This is obviously not acceptable, and if the list of cached files is incorrect for the version it will be even worse.

  • Now imagine the web app is updated in this manner:

    1. remainder is updated
    2. manifest is updated 30 seconds (one network timeout) later

    In this case, the scenario in the previous case cannot occur: if the user fetched the HTML resource prior to the update, the user agent will either succeed before the manifest is updated, or will give up at its network timeout. However, another scenario can now occur:

    1. remainder is updated
    2. user loads the app from the server (either initial install or because he still had a version prior to the one before the update), both app files an manifest
    3. manifest is updated

    In that case, the user has the updated app but the manifest for the previous version. Even if the list of cached files is correct, the versions are inconsistent which is an issue if the new version turns out to have a showstopping issue (which sometimes only becomes apparent after public deployment, due to the enormous variety of user agents in the wild) and we decide to roll back to the previous version: in that case, whenever the user agent checks for an updated manifest, it will find it hasn’t changed and the user will keep using the version of the app that has a showstopping issue. When performing the rollback, we could decide to modify the manifest so that it is different from both versions, but this is dangerous: when rolling back you want to deploy exactly what you deployed before, in order to avoid running into further issues. And I don’t need to tell you how problematic having inconsistent app and manifest would be if the list of resources to cache changed during the update.

So how does Curtain solve this problem?

By updating the manifest twice:

  1. manifest is updated with intermediate contents
  2. remainder is updated
  3. manifest is updated again 30 seconds (one network timeout) later

If the list of resources to cache changes during the update, the manifest contains the union of the files needed by the previous version and the files needed by the updated version; and in all cases, the intermediate manifest contains in a comment two version numbers: the one for the app prior to the update, and the one for the app after the update. That way the manifest is suitable in both cases, and his method of update avoid all the issues associated with the previous methods.

Of course, that would be tedious and error-prone to handle by hand, so Curtain generates both intermediate and updated manifests from a script.

Versioned resources

I enjoy reading Clients from Hell; even though I don’t design web sites for a living I relate strongly to these horror stories. Except for one kind: those where the client complains he should not have to clear the cache/do a hard reload/etc. to see the fully updated site. Sorry, but for those, I side completely and unquestioningly with the client. Even in a development iteration context, it is up to the developer to show he can change the site and have the changes propagate without the user needing to do anything more than a soft reload (which invalidates the initial HTML resource if necessary, but nothing else), because such changes will need to happen in a post-deployment context. And don’t get me started on the number of site redesigns where the previous versions of all assets (icons, previous/next arrows, etc.) are still visible, and the announcement post starts with the caveat that you may have to reload manually in order for the redesign to be fully in effect… and even then, it has to be done again on a second page, because the main page does not have a “next” arrow for instance.

Yes, clearly you want resources and image assets, in particular, to be far-expire in order to save on bandwidth. But this means they must also be immutable: they might disappear, but may never, ever change; and if a different resource is needed, then it must have a different URL. Period.

Obviously, changing the resource name by hand, especially if you need to do so for every development iteration, is tedious and error-prone. When I read in web development tutorials, including some Application Cache ones, the suggestion to use, say, script-v2.js and increment version numbers that way, I can’t help but think “Isn’t that the cutest thing? Thinking you can do so flawlessly without ever forgetting to so do whenever a resource changes? Awwww…” because that is a recipe for failure, even if you only change these resources as part of a deploy.

Such inconsistency issues are even worse for offline web apps. Indeed, if your web app cannot work offline, you can just assume that, if your web app works incorrectly because of an inconsistent set of resources, the user will just reload and she will eventually get consistent resources. But in the case of an offline web app, once the user is back to her new home for which DSL hasn’t been installed yet (I’m getting tired of the airplane example) she has no opportunity for reload.

Even worse, even if the user checked while she was online that the web app was working correctly (which is asking a lot of her already), it may in fact be the previous version that was reloaded from the cache, while an inconsistently updated version is being downloaded, and when she relaunches it while at home she will get the inconsistent version. You can’t afford to be careless with offline web apps.

Curtain resolves this issue by relying on a version control system. On the build machine, all resources must be under a version-controlled work area, and Curtain will query the version control system for the ID of the version where the resource was last updated, and will generate a resource name by appending this version ID. Note that by doing it this way, Curtain will avoid changing the URL of the resource (which would invalidate it in the cache) even if everything else has changed, as long as the resource itself hasn’t changed. Curtain will process your HTML to replace references to the resource by references to the versioned resource name, and upload the result, and upload the resources themselves so that they have the versioned name on the server.

Curtain will also assign a version to the app as a whole, this is in particular put in a comment in the manifest (see above): this version is simply the current version of the version control system work area. Curtain itself must be in such a work area, so that if Curtain is updated but the source files are not, the version number is changed.

As part of these tasks, Curtain will manage the Cache-Control headers by synthesizing the necessary .htaccess file, which is especially important when using Application Cache; since it has to deal with .htaccess anyway, Curtain will also directly manage the MIME type of these resources, to avoid relying on the default Apache behavior (based on file extensions).

No progressive rendering

I have always found progressive rendering to be unsightly. It was necessary in the first days of the web, what with images taking seconds to download, it is largely necessary on mobile to this day, and it is still desirable on desktop for online apps. But for offline, desktop-like web apps? No way.

Curtain opts out of progressive rendering by downloading all dependent resources through XMLHttpRequest and explicitly loading the content, for instance for image resources by generating a URL for the downloaded Blob and assigning it through code to the src attribute of the img tag; this means Curtain-deployed web apps depend on XHR2 and Blob as a XHR responseType. Curtain will hide the interface until all resources have been loaded and assigned, assuming that the user will retry loading the app if no interface appears after a time; it is safe to assume the user is online at that time, because if he is offline, this means all the files listed in the Application Cache manifest are locally available and so will not fail to load.

If JavaScript is disabled or the browser does not have the necessary support for Curtain, we want to be able to show a message to that effect, and we want to do it in the context of the “usual” interface, so that she recognizes the web app. So the entirety of the interface is put in a div belonging to a CSS class called curtain. A small bit of JavaScript code before the interface hides this div: if JavaScript is disabled, the interface simply won’t be hidden. Then code after the interface will check everything necessary for the Curtain runtime to perform its job (using Modernizr in particular); if not everything is available, then the message will be changed and the div will be made visible.

The HTTP URLs of the images are put in the src attributes of the img tags in the initially downloaded HTML. However, this is only a provision for the above two error cases; in normal usage they will have been replaced by the blob URLs prior to the interface becoming visible.

Language

First, Curtain generates static sites, and does not depend on any server programming language or any kind of server processing. Second, while early versions of the build and upload script were written as a shell script, Curtain is written in Python so as to be as portable as possible (it was that or Perl; I chose Python), though I have not been able to test it on Windows or Linux yet.

Third, Curtain embeds a bit of JavaScript code along with your app, and it expects your app to be written in JavaScript. However, Curtain makes no pretense at bringing JavaScript framework features; you should be able to use it with any JavaScript framework, including Vanilla JS.

Stay tuned…

Stay tuned, because tomorrow I will present you the sample app for Curtain, and its justification.

Various comments on the iPad pro, among other Apple announcements

As someone who regularly expresses himself on how the iPad is (or how it currently falls short of being) the future of computing, I guess I should comment on the announcements Apple made this September, in particular of course the iPad pro. I think it’s great news, of course. But… where to begin? More so than the iPhone, it is software applications, and specifically third-party software, that will make or break the iPad, even more so for the iPad pro, considering that most of its hardware improvements will only really be exploited with third-party apps: it does not appear that Apple will provide a built-in drawing app to go with the impressive tablet drawing capabilities of the iPad pro, for instance.

And so, what corresponding iOS platform news did we get from Apple this September? Err, none. From a policy standpoint, iOS is still as developer-unfriendly, by not supporting software trials for instance, even though this is a fundamental commercial practice; in fact, this appear to be backfiring on Apple, as this resulted in potential buyers going for the already established brands1 when it comes to professional iPad software, and in Apple coming to bringing Adobe and Microsoft on stage in the presentation to prove the professional potential of the iPad pro; those two companies are probably the ones Apple wishes to be least dependent upon, and yet here we are.

And what about iOS, iOS 9 specifically? When it was announced at WWDC, my thoughts were, on the timing: “It’s about time!”, and on the feature set: “This is a good start.”; and this was in the context of the 10′ iPad, so with the iPad pro announcement I was expecting more multitasking features to be announced along with it, but nope, that was it, what I saw at WWDC and thought would be catching up, meant for the then-current iPads, was in fact a stealth preview of software features meant to be sufficient for the iPad pro. Don’t get me wrong, on-screen multitasking on the iPad is awesome (I’ve been playing with it this summer with the developer betas), but the iPad pro will need more than that. Much more, such as, I don’t know, drag and drop between apps? Or a system where one app could ask a service of another other app (bringing it on the screen by the first app’s side if it wasn’t present), similar to extensions except it would better use the screen real estate allowed by the iPad and iPad pro?

So yes, most definitely count me in with those who think Apple is solving only one part of the problem of the iPad platform with the iPad pro. I’m afraid the iPad has too little direct competition for Apple to take this problem seriously right now, and later it may be too late.

All that having been said, I am very impressed with what they’ve shown of the iPad pro’s capabilities as a graphic tablet, even if I will probably never use these capabilities myself; I think Wacom should indeed be worried. One thing that will matter a lot, but Apple hasn’t talked about or posted specs on, is the thinness of the so-called air gap between the screen and the drawing surface, which (along with latency, which they did mention in the keynote) makes all the difference in making the user feel like he is actually drawing on a surface, rather than awkwardly interacting with a machine; Piro of Megatokyo posted about this a while ago. This is something Apple has been good at, at least when compared with other handset/tablet makers. At any rate, the iPad, even the iPad pro, has to support the needs of everyone, so Wacom and other graphic tablet makers may still be able to differentiate themselves by providing possibilities that a tablet that still must support finger input and other usage patterns than drawing could not provide.

Lastly, I am interested in the iPad pro as a comic reader. The 10′ iPad family is generally great for reading manga and U.S. comics, as that screen size is large enough to contain a single page of them (in the case of U.S. comics, pages are actually slightly scaled down, but that does not impair reading), however it falls short when needing to show a double page spread, such as the (outstanding) one found in the latest issue of Princeless (if you’re not, you should be reading Princeless. Trust me.) or the not less outstanding ones found in Girl Genius. The iPad pro has the perfect size and aspect ratio for those, being able of showing two U.S. comics pages side by side at the same scale a 10′ iPad is able of showing a single page. A 10′ iPad is also too small to reasonably display a single page of comics of the French-Belgian tradition (aka “bandes dessinées”), while the iPad pro would be up to the task with only a minor downscale; I’m not about to give up buying my French-Belgian comics on paper any time soon, but it would be a wonderful way to read them for someone overseas who is far away from any store that distributes such comics, especially for less well-known ones.2

As for the other announcements… I don’t know what the new Apple TV will be capable of outside the U.S., so most of its appeal is hard to get for me. I have little to comment on with regard to its capabilities as an app platform, except that I find the notion of Universal iOS/tvOS games (even more so saved game handoff) to be completely ludicrous: the two have completely different interaction mechanisms, so the very notion that the “same game” could be running on both is absurd.

As for the iPhone 6S/6S+, what is to me most interesting about them is 3D touch and live photos. Those two are very dependent on the concept catching on with developers (well, Vine is ready for live photos, of course…): there is little point in being able to capture live photos if you can’t share them as such on Facebook/Flickr/Dropbox/etc., and it remains to be seen which developers will add 3D touch actions to their apps, and what for. So will they catch on? We’ll see.


  1. And they, in turn, have the branding power to pull off alternate payment schemes such as subscriptions, allowing them to thrive on the iPad more than developers dependent on race-to-the-bottom iOS App Store revenue do.
  2. What about double page spreads in French-Belgian comics? Those are rare, but they exist, most notably in the œuvre of Cabu (yes, that Cabu). However, I am not sure the 17′ handheld tablet that would be necessary to properly display these would be very desirable.

Maybe Android did have the right idea with activities

One of the features that will land as part of iOS 9 is a new way to display web content within an app by way of a new view controller called SFSafariViewController, which is nicely covered by Federico Viticci in this article on MacStories. I’ve always been of the (apparently minority) opinion that iOS apps ought to open links in the default browser (which currently is necessarily Mobile Safari), but the Safari view controller might change my mind, because it pretty much removes any drawback the in-app UI/WKWebView-based browsers had: with the Safari view controller I will get a familiar and consistent interface (minus the editable address bar, of course), the same cookies as when I usually browse, my visits will be registered in the history, plus added security (and performance, though WKWebView already had that), etc. Come November, maybe I will stop long-tapping any and all links in my Twitter feed (in Twiterrific, of course) in order to bring the share sheet and open these links in Safari.

Part of my reluctance to use the in-app browsers was that I considered myself grown up enough to switch apps, then switch back to Twitter/Tumblr/whatever when I was done with reading the page: I have never considered app switching to be an inconvenience in the post-iOS 4 world, especially with the iOS 7+ multitasking interface. But I understand the motivation behind wanting to make sure the user goes back to whatever he was doing in the app once he is done reading the web content; and no, it is not about the app developers making sure the user does not stray far from the app (okay, it is not just about that): the user himself may fail to remember that he actually was engaged in a non-web-browsing activity and thus is better served, in my opinion, by a Safari experience contained in-app than he is by having to switch to Safari. For instance, if he is reading a book he would appreciate resuming reading the book as soon as he is done reading the web page given as reference, rather than risk forgetting he was reading a book in the first place and realize it the following day as he reopens the book app (after all, there is no reason why only John Siracusa ebooks should have web links).

And most interestingly, in order to deliver on all these features, the Safari view controller will run in its own process (even if, for app switching purposes for instance, it will still belong to the host app); this is a completely bespoke tech, to the best of my knowledge there is no framework on iOS for executing part of an app in a specific process as part of a different host app (notwithstanding app extensions, which provide a specific service to the host app, rather than the ability to run a part of the containing app). And this reminded me of Android activities.

For those of you not familiar with the Android app architecture, Android applications are organized around activities, each activity representing a level of full-screen interaction, e.g. the screen displaying the scrollable list of tweets complete with navigation bars at the top and/or bottom, with the activity changing when drilling down/back up the app interface, e.g. when tapping a tweet the timeline activity slides out and the tweet details activity slides in its place, with activities organized in a hierarchy. Android activities are roughly equivalent to view controllers on iOS, with a fundamental difference: installed apps can “vend” activities, that other apps can use as part of their activity hierarchy. For instance, a video player app can start with an activity listing the files found in its dedicated folder, then once such a file is tapped it starts the activity to play a single video, all the while also allowing any app, such as an email client or the file browser, that wants to play a video the ability to directly invoke the video-playing activity from within the host app.

I’ve always had reservations with this system: I thought it introduced user confusion as to which app the user “really” is in, which matters for app switching and app state management purposes, for instance. And I still do think so to an extent, but it is clear the approach has a lot of merit when it comes to the browser, given how pervasive visiting web links is even as part of otherwise very app-centric interactions. And maybe it could be worth generalizing and creating on iOS an activity-like system; sure, the web browser is pretty much the killer feature for such a system (I am more guarded about the merits when applying it to file viewing/editing, as in my video player example), but other reasonable applications could be considered.

So I’m asking you, would you like, as a user, to have activity-like features on iOS? Would you like, as a developer, to be able to provide some of your app’s view controllers to other apps, and/or to be able to invoke view controllers from other apps as part of your app?

“Character”-by-“character” string processing is hard, people

I bet you did not believe me when I wrote in Swift thoughts about how it is hard to properly process strings when treating them as a sequence of Unicode code points, and that as a result text is better thought of as a media flow and strings better handled through the few primitives I mentioned, which never treat strings using any individual entity (be this entity the byte, the UTF-16 “character”, the Unicode code point, or the grapheme cluster). I am exaggerating, of course, some of you probably did believe me, but given how I still see string processing being discussed between software developers, this is true enough.

So go ahead and read the latest post in the Swift blog, about how they changed the String type in Swift 2, and the fact that it is no longer considered a collection (by no longer conforming to the CollectionType protocol), because a collection where appending an element (a combining acute accent, or “´”) not only does not result in the element being considered part of the collection, but also results in elements previously considered part of it (the letter “e”) to no longer be, is a pretty questionable collection type. Oops.

But that is not the (most) interesting aspect of that blog post.

Look at the table towards the end, which is supposed to correspond to a string “comprised of the decomposed characters [ c, a, f, e ] and [ ´ ]”, and which I am reproducing here, as an actual HTML table as Tim Berners-Lee intended, for your benefit (and because I am pretty certain they are going to correct it after I post this):

Character

c

LATIN SMALL LETTER C
U+0063

a

LATIN SMALL LETTER A
U+0061

f

LATIN SMALL LETTER F
U+0066

é

LATIN SMALL LETTER E WITH ACUTE
U+00E9

Unicode Scalar Value

c

a

f

e

´

UTF-8 Code Unit

99

97

102

101

204

UTF-16 Code Unit

99

97

102

769

The first thing you will notice is the last element of the Character view, the whole row in fact. Why are they described by a Unicode code point each? Indeed, each of these elements is an instance of the Swift Character type, i.e. a grapheme cluster, which can be made up of multiple code points, and this is particularly absurd in the case of the last one, which corresponds to two Unicode code points. True, it would compare equal with a Swift Character containing a single LATIN SMALL LETTER E WITH ACUTE, but that is not what it contains. And yet, this is only the start of the problems…

If we take the third row, its last element is incorrect. Indeed, 204, or 0xCC ($CC for the 68k assembly fans in the audience) is only the first byte of the UTF-8 serialization of U+0301 (COMBINING ACUTE ACCENT) that you see in the previous row (which is correct, amazingly), the second being $81.

And lastly, if the last two column are two separate Unicode scalar values, how could they possibly be represented by a single UTF-16 scalar? Of course, they can’t: 769 is $0301, our friend the combining acute accent. “e” is simply gone.

So out of 4 rows, 3 are wrong *slow clap*. So here is the correct table:

Character

c

a

f

Unicode Scalar Value

c

LATIN SMALL LETTER C
U+0063

a

LATIN SMALL LETTER A
U+0061

f

LATIN SMALL LETTER F
U+0066

e

LATIN SMALL LETTER E
U+0065

´

COMBINING ACUTE ACCENT
U+0301

UTF-8 Code Unit

$63

$61

$66

$65

$CC

$81

UTF-16 Code Unit

$0063

$0061

$0066

$0065

$0301

Note that with the example given, Unicode scalar value match one for one with the UTF-16 scalars in the sequence. For a counterexample to be provided, the string would have to include Unicode code points beyond the Basic Multilingual Plane — a land populated by scripts no longer in general usage (hieroglyphs, Byzantine musical notations, etc.), extra compatibility ideographs, invented languages, and other esoteric entities; that place, by the way, is where emoji were (logically) put in Unicode.

Conclusion

If Apple can’t get its “Characters”, UTF-16 scalars, and bytes of a seemingly simple string such as “café” straight in a blog post designed to show these very views of that string, what hope could you possibly have of getting “character”-wise text processing right?

Treat text as a media flow, by only using string processing primitives without ever directly caring about the individual constituents of these strings.

Thank you, Mr. Siracusa

Today, I learned that John Siracusa had retired from his role of writing the review of each new Mac OS X release for Ars Technica. Well, review is not quite the right word: as I’ve previously written when I had the audacity to review one of his reviews, what are ostensibly articles reviewing Mac OS X are, to my mind, better thought of as book-length essays that aim to vulgarize the progress made in each release of Mac OS X. They will be missed.

It would be hard for me to overstate the influence that John Siracusa’s “reviews” have had on my understanding of Mac OS X and on my writing; you only have to see the various references to John or his reviews I made over the years on this blog (including this bit…). In fact, the very existence of this blog was inspired in part by John: when I wrote him with some additional information in reaction to his Mac OS X Snow Leopard review, he concluded his answer with:

You should actually massage your whole email into a blog post [of] your own.  I’d definitely tweet a link to it! 🙂

to which my reaction was:

Blog? Which blog? On the other hand, it’d be a good way to start one
Hmm

Merely 4 months later, for this reason and others, this blog started (I finally managed to drop the information alluded to in 2012; still waiting for that tweet 😉 ).

And I’ll add that his podcasting output may dwarf his blogging in volume, but, besides the fact I don’t listen to podcasts much, I don’t think they really compare, mostly because podcasts lack the reference aspect of his Mac OS X masterpieces due to the inherent limitations of podcasts (not indexed, hard to link to a specific part, not possible to listen in every context, etc.). But, ultimately, it was his call; as someone, if I remember well, commented on the video of this (the actual video has since gone the way of the dodo): “Dear John, no pressure. Love, the Internet”. Let us not mourn, but rather celebrate, from the Mac OS X developer preview write-ups to the Mac OS X 10.10 Yosemite review, the magnum opus he brought to the world. Thank you, Mr. Siracusa.

Unconventional iOS app extension idea: internal thumbnail generator

The arrival (along with similar features) of extensions in iOS 8, even if it does not solve all problems with the platform’s inclusiveness, represents a sea change in what is possible for third-party developers with iOS, enabling many previously unviable apps such as Transmit iOS. But, even with the ostensibly specific scenarios (document provider extensions, share extensions, etc.) app extensions are allowed to hook themselves to, I feel we have only barely begun to realize the potential of extensions. Today I would like to present a less expected problem extensions could solve: fail-safe thumbnail generation.

The problem is one we encountered back in the day when developing CineXPlayer. I describe the use case in a radar (rdar://problem/9115604), but the tl;dr version is we wanted to generate thumbnails for the videos the user loaded in the app, and were afraid of crashing at launch as a result of doing this processing (likely resulting in the user leaving a one-star “review”), so we wanted to do so in a separate process to preserve the app, but the sandbox on iOS does not allow it.

But now in iOS 8 there may be a way to use extensions to get the same result. Remember that extensions are run in their own process, separate from both the host app process and the containing app process; so the idea would be to embed an action extension for a custom type of content that in practice only our app provides, make the videos loaded in the app provided under that type to extensions, and use the ability of action extensions to send back content to the host to send back the generated thumbnail; if our code crashes while generating the thumbnail, we only lose the extension process, and the app remains fine.

This would not be ideal, of course, as the user would have to perform an explicit action on each and every file (I haven’t checked to see whether there would be sneaky ways to process all files with one extension invocation), but I think it would be worth trying if I were still working on CineXPlayer; and if after deployment Apple eventually wises up to it, well, I would answer them that it’s only up to them to provide better ways to solve this issue.

MPW on Mac OS X

From Steven Troughton-Smith (via both Michael Tsai and John Gruber) comes the news of an MPW compatibility layer project and how to use it to build code targeting Classic Mac OS and even Carbonized code from a Mac OS X host, including Yosemite (10.10). This is quite clever, and awesome news, as doing so was becoming more and more complicated, and in practice required keeping one ore more old Macs around.

Back in the days of Mac OS X 10.2-10.4, I toyed with backporting some of my programming projects, originally developed in Carbon with Project Builder, to MacOS 9, and downloaded MPW (since it was free, and CodeWarrior was not) to do so. The Macintosh Programmer’s Workshop was Apple’s own development environment for developing Mac apps, tracing its lineage from the Lisa Programmer’s Workshop, which was originally the only way to develop Mac apps (yes, in 1984 you could not develop Mac software on the Mac itself). If I recall correctly, Apple originally had MPW for sale, before they made it free when it could no longer compete with CodeWarrior. You can still find elements from MPW in the form of a few tools in today’s Xcode — mostly Rez, DeRez, GetFileInfo and SetFile. As a result, I do have some advice when backporting code from Mac OS X to MacOS 9 (and possibly earlier, as Steven demonstrated).

First, you of course have to forget about Objective-C, forget about any modern Carbon (e.g. HIObject, though the Carbon Event Manager is OK), forget about Quartz (hello QuickDraw), forget about most of Unix, though if I recall correctly the C standard library included with MPW (whose name escapes me at the moment) does have some support beside the standard C library, such as open(), read(), write() and close(). Don’t even think about preemptive threads (or at least, ones you would want to use). In fact, depending on how far back you want to go, you may not have support for what you would not even consider niceties, but were actually nicer than what came before; for instance, before Carbon, a Mac app would call WaitNextEvent() in a loop to sleep until the next event that needed processing, and then the app would have to manually dispatch it to the right target, including switching on the event type, performing hit testing, etc.: no callback-based event handing! But WaitNextEvent() itself did not appear until System 7, if I recall correctly, so if you want to target System 6 and earlier, you have to poll for events while remembering to yield processing time from time to time to drivers, to QuickTime (if you were using it), etc. The same way, if you want to target anything before MacOS 8 you cannot use Navigation Services and instead have to get yourself acquainted with the Standard File Package… FSRefs are not usable before MacOS 9, as another example.

When running in MacOS 9 and earlier, the responsibilities of your code also considerably increase. For instance, you have to be mindful of your memory usage much more than you would have to in Mac OS X, as even when running with virtual memory in MacOS 9 (something many users disabled anyway) your application only has access to a small slice of address space called the memory partition of the application (specified in the 'SIZE' resource and that the user can change): there is only one address space in the system which is partitioned between the running apps; as a result memory fragmentation becomes a much more pressing concern, requiring in practice the use of movable memory blocks and a number of assorted things (move high, locking the block, preallocating master pointers, etc.). Another example is that you must be careful to leave processor time for background apps, even if you are a fullscreen game: otherwise, for instance if iTunes is playing music in the background, it will keep playing (thanks to a trick known as “interrupt time”)… until the end of the track, and become silent from then on. Oh, and did I mention that (at least before Carbon and the Carbon Event Manager) menu handling runs in a closed event handling loop (speaking of interrupt time) that does not yield any processing time to your other tasks? Fun times.

Also, depending again on how far back you want to go, you might have difficulty using the same code in MacOS 9 and Mac OS X, even with Carbon and CarbonLib (the backport of most of the Carbon APIs to MacOS 9 as a library, in order to support the same binary and even the same slice running on both MacOS 9 and Mac OS X). For instance, if you use FSSpec instead of FSRef in order to run on MacOS 8, your app will have issues on Mac OS X with file names longer than were possible on MacOS 9; they are not fatal, but will cause your app to report the file name as something like Thisisaverylongfilena#17678A… not very user-friendly. And the Standard File Package is not supported at all in Carbon, so you will have to split your code at compile time (so that the references to the Standard File Package are not even present when compiling for Carbon) and diverge at runtime so that when running in System 7 the app uses the Standard File Package, and when running in MacOS 8 and later it uses Navigation Services, plus the assorted packaging headaches (e.g. using a solution like FatCarbon to have two slices, one ppc that links to InterfaceLib, the pre-Carbon system library, linking weakly to the Navigation Services symbols, and one ppc that links to CarbonLib and only runs on Mac OS X).

You think I’m done? Of course not, don’t be silly. The runtime environment in MacOS 9 is in general less conductive to development than that of Mac OS X: the lack of memory protection not only means that, when your app crashes, it is safer to just reboot the Mac since it may have corrupted the other applications, but also means you typically do not even know when your code, say, follows a NULL pointer, since that action typically doesn’t fault. Cooperative multitasking also means that a hang from your app hangs the whole Mac (only the pointer is still moving), though that can normally be solved by a good command-alt-escape… after which it’s best to reboot anyway. As for MacsBug, your friendly neighborhood debugger… well, for one, it is disassembly only, no source. But you can handle that, right?

It’s not that bad!

But don’t let these things discourage you from toying with Classic MacOS development! Indeed, doing so is not as bad as you could imagine from the preceding descriptions: none of those things matter when programming trivial, for fun stuff, and even if you program slightly-less-than-trivial stuff, your app will merely require a 128 MB memory partition where it ought to only take 32 MB, which doesn’t matter in this day and age.

And in fact, it is a very interesting exercise because it allows a better understanding of what makes the Macintosh the Macintosh, by seeing how it was originally programmed for. So I encourage you all to try and play with it.

For this, I do have some specific advice about MPW. For one, I remember MrC, the PowerPC compiler, being quite anal-retentive for certain casts, which it just refuses to do implicitly: for instance, the following code will cause an error (not just a warning):

SInt16** sndHand;
sndHand = NewHandle(sampleNb * sizeof(SInt16));

You need to explicitly cast:

SInt16** sndHand;
sndHand = (Sint16**)NewHandle(sampleNb * sizeof(SInt16));

It is less demanding when it comes to simple casts between pointers. Also, even though it makes exactly no difference in PowerPC code, it will check that functions that are supposed to have a pascal attribute (supposed to mark the function as being called using the calling conventions for Pascal, which makes a difference in 68k code), typically callbacks, do have it, and will refuse to compile if this is not the case.

If you go as far back as 68k, if I remember correctly int is 16 bit wide in the Mac 68k environment (this is why SInt32 was long up until 64-bit arrived: in __LP64__ mode SInt32 is int), but became 32 bit wide when ppc arrived, so be careful, it’s better not to use int in general.

QuickDraw is, by some aspects, more approachable that Quartz (e.g. no object to keep track of and deallocate at the end), but on the other hand the Carbon transition added some hoops to jump through that makes it harder to just get started with it; for instance something as basic as getting the black pattern, used to ensure your drawing is a flat color, is described in most docs as using the black global variable, but those docs should have been changed for Carbon: with Carbon GetQDGlobalsBlack(&blackPat); must be used to merely get that value. Another aspect which complicates initial understanding is that pre-Carbon you would just directly cast between a WindowPtr, (C)GrafPtr, offscreen GWorldPtr, etc., but when compiling for Carbon you have to use conversion functions, for instance GetWindowPort() to get the port for a given window… but only for some of those conversions, the others just being done with casts, and it is hard to know at a glance which are which.

When it came to packaging, I think I got an app building for classic MacOS relatively easily with MPW, but when I made it link to CarbonLib I got various issues related to the standard C library, in particular the standard streams (stdin, stdout and stderr), and I think I had to download an updated version of some library or some headers before it would work and I could get a single binary that ran both in MacOS 9 and natively on Mac OS X.

Also, while an empty 'carb' resource with ID 0 does work to mark the application as being carbonized and make it run natively on Mac OS X, you are supposed to instead use a 'plst' resource with ID 0 and put in there what you would put in the Info.plist if the app were in a package. Also, it is not safe to use __i386__ to know whether to use framework includes (#include <Carbon/Carbon.h>) or “flat” includes (#include <Carbon.h>); typically you’d use something like WATEVER_USE_FRAMEWORK_INCLUDES, which you then set in your Makefile depending on the target.

Lastly, don’t make the same mistake I originally did: when an API asks for a Handle, it doesn’t just mean a pointer to pointer to something, it means something that was specifically allocated with NewHandle() (possibly indirectly, e.g. with GetResource() and loaded if necessary), so make sure that is what you give it.

I also have a few practical tips for dealing with Macs running ancient system software (be they physical or emulated). Mac OS X removed support for writing to an HFS (as opposed to HFS+) filesystem starting with Mac OS X 10.6, and HFS is the only thing MacOS 8 and earlier can read. However, you can still for instance write pre-made HFS disk images to floppy discs with Disk Utility (and any emulator worth its salt will allow you to mount disk images inside the emulated system), so your best bet is to use a pre-made image to load some essential tools, then if you can, set up a network connection (either real or emulated) and transfer files that way, making sure to encode them in MacBinary before transfer (which I generally prefer to BinHex); unless you know the transfer method is Mac-friendly the whole way, always decode from MacBinary as the last step, directly from the target. Alternately, you can keep around a Mac running Leopard around to directly write to HFS floppies, as I do.

Okay, exercise time.

If you are cheap, you could get away with only providing a 68k build and a Mac OS X Intel build (except neither of these can run on Leopard running on PowerPC…). So the exercise is to, on the contrary, successfully build the same code (modulo #ifdefs, etc.) for 68k, CFM-PPC linking to InterfaceLib, CFM-PPC linking to CarbonLib, Mach-o Intel, Mach-o 64-bit PPC, and Mach-o 64-bit Intel (a Cocoa UI will be tolerated for those two) for optimal performance everywhere (ARM being excluded here, obviously). Bonus points for Mach-o PPC (all three variants) and CFM-68k. More bonus points for gathering all or at least most of those in a single obese package.

Second exercise: figure out the APIs which were present in System 1.0 and are supported in 64-bit on Mac OS X. It’s a short list, but I know for sure it is not empty.

References

Macintosh C Carbon: besides the old Inside Mac books (most of which can still be found here), this is how I learned Carbon programming back in the day.

Gwynne Raskind presents the Mac toolbox for contemporary audiences in two twin articles, reminding you in particular to never neglect error handling, you can’t get away with it when using the toolbox APIs.

And on a lighter note about Swift…

One more thought on the matter of Swift, which wasn’t suitable for my previous post (and is too long for Twitter)

2014 – Chris Lattner invents Swift. Swift is an admittedly relatively concise, automatically reference counted, but otherwise class based, statically typed, single dispatch, object oriented language with single implementation inheritance and multiple interface inheritance. Apple loudly heralds Swift’s novelty.

(With apologies to James Iry, and to William Ting, who beat me to it except he mischaracterized Swift as being garbage collected.)