Software Reenchantment

I’ve had Nikitonsky’s Software Disenchantment post in my mind ever since it was posted… which was four months ago. It’s fair to say I’m obsessed and need to get the following thoughts out of my system.

First because it resonated with me, of course. I recognize many of the situations he describes, and I share many of his concerns. There is no doubt that many evolutions in software development seem incongruous and at odds with recommendations for writing reliable software. The increasing complexity of the software stack, in particular, is undoubtedly a recipe for bugs to thrive, able to hide in that complexity.

Yet some of that complexification, even controversial, is nevertheless a progress. The example that comes to mind is Chrome and more specifically its architecture of running each tab (HTML rendering, JavaScript, etc.) on its own process for reliability and security, and the related decision to develop a high-performance JavaScript engine, V8, that dynamically compiles JavaScript to native code and runs that (if you need a refresher, Scott McCloud’s comic is still relevant). Yes, this makes Chrome a resource hog, and initially I was skeptical about the need: the JavaScript engine controls the generated code, so if it did its work correctly, it would properly enforce same-origin and other restrictions, without the need of the per-tab process architecture and its overhead of creation, memory occupation, etc. of many shell processes.

But later on I started seeing things differently. It is clear that browser developers have been for the last few years engaged in a competition for performance, features, etc., even if they don’t all favor the same benchmarks. In that fast-paced environment, it would be a hard dilemma between going for features and performance at the risk of bugs, especially security vulnerabilities, slipping through the cracks, and instead moving at a more careful pace, at the risk of being left behind by more innovative browsers and being marginalized; and even if your competitor’s vulnerabilities end up catching up with him in the long term, that still leaves enough time for your browser to be so marginalized that it cannot recover. We’re not far from a variant of the prisoner’s dilemma. Chrome resolved that dilemma by going for performance and features, and at the same time investing up front in an architecture that provides a safety net so that a single vulnerability doesn’t mean the attacker can escape the jail yet, and bugs of other kinds are mitigated. This frees the developers working on most of the browser code, in particular on the JavaScript engine, from excessively needing to worry about security and bugs, with the few people having the most expertise on that instead working on the sandbox architecture of the browser.

So that’s good for the browser ecosystem, but the benefits extend beyond that: indeed the oneupmanship from this competition will also democratize software development. Look, C/C++ is my whole carrier, I love system languages, there are many things you can do only using them even in the applicative space (e.g. real-time code such as for A/V applications), and I intend to work in system languages as long as I possibly can. But I realize these languages, C/C++ in particular, have an unforgiving macho “it’s your fault you failed, you should have been more careful” attitude that makes them unsuitable for most people. Chrome and the other high-performance browsers that the others have become since then vastly extend the opportunities of JavaScript development, with it starting now to be credible for many kinds of desktop-like applications. JavaScript has many faults, but it is also vastly more forgiving than C/C++, if only by virtue of it providing memory safety and garbage collection. And most web users can discover JavaScript by themselves with “view source”. Even if C/C++ isn’t the only game in town for application development (Java and C# are somewhat more approchable, for instance), this nevertheless removes quite a hurdle to starting application development, and this can only be a good thing.

And of course, the per-tab process architecture of Chrome really means it ends up piggybacking on the well-understood process separation mechanism of the OS, itself relying of the privilege separation feature of the processor, and after meltdown and spectre it would seem this bedrock is more fragile than we thought… but process separation still looks like a smart choice even in this context, as a long-term solution will be provided in newer processors for code running in different address spaces (at the cost of more address space separation, itself mitigated by features such as PCID), while running untrusted code in the same address space will have no such solution and is going to become more and more of a black art.

So I hope that, if you considered Chrome to be bloated, you realize now it’s not so clear-cut. So more complexity can be a good thing. On the other hand, I have an inkling that the piling on of dependencies in the npm world in general, and in web development specifically, is soon going to be unsustainable, but I’d love to be shown wrong. We need to take a long, hard look at that area in general.

So yes, it’s becoming harder to even tell if we software engineers can be proud of the current state of our discipline or not. So what can be done to make the situation clearer, and if necessary, improve it?

First, we need to treat software engineering (and processor engineering, as we’re at it) just like any other engineering discipline, by having software developers need to be licensed in order to work on actual software. Just like a public works engineer needs to be licensed before he can design bridges, a software engineer will need to be licensed before he can design software that handles personal data, with the requirement repeating down to the dependencies: for this purpose, only library software that has itself been developed by licensed software engineers could be used. We would need to grandfather in existing software, of course, but this is necessary as software mistakes are (generally) not directly lethal, but can be just as disruptive to society as a whole when personal data massively leaks. Making software development require a license would in particular provide some protection against pressure from the hierarchy and other stakeholders such as marketing, a necessary prerequisite enabling designers to say “No” to unethical requests.

Second, we need philosophers (either comings from our ranks or outsiders) taking a long hard look at the way we do things and trying to make sense of it, to even figure out the questions that need asking for instance, so that we can be better informed of where we as a discipline need to work on. I don’t know of anyone right now doing this very important job.

These, to me, are the first prerequisites to an eventual software reenchantment.

Factory Hiro on iPad is the real deal

I tested for you Factory Hiro, the latest attempt at remaking Factory: The Industrial Devolution on iPad (and iPhone, though I tested it on iPad). After the previous attempt towards that goal (if it could even be called an attempt), caution was certainly warranted, after all. So is it worth it?

The short answer: yes. Factory Hiro is the real deal, buy and download it without fear, it is a worthy remake of the original, finally available with a touch interface where it can shine.

The longer answer: compared to the original, graphics have obviously been remade, but everything in the gameplay will otherwise be familiar. Redirection boxes that you tap to toggle between vertical and horizontal direction, the trash destination and the recycle stream (and of course the default final destination: the delivery truck), assembling steps that you sometimes have to turn on or off, etc. It’s all there. In particular, the regulation of assembly line speed so you get your quota done in time for the end of the day, but not so fast that you end up being overwhelmed by the oncoming components to manage, is still the fundamental challenge of the game.

A nicety was added, though: when some trash gets generated (“Oh No!”), the speed will automatically switch to slowest for you, so it is much easier to manage these crises when they come. Other differences exist, but are less significant.

In non-gameplay aspects, the story was changed as well; these days of course it is told as cutscenes (created by KC Green) depicting the titular Hiro proving his hierarchy that yes, he can get the job done. And you, will you succeed?

Factory Hiro is available on the iOS App Store, as well as on the Google Play store for the Android version, and on PC and Mac through Steam (I got it for 3.49€ on the French iOS App Store; pricing will depend on your region); it was reviewed on an iPad Air 2 running iOS 11.4.1.

P.S.: In similar nostalgia-inspired discoveries, I should mention that I can’t believe I went such a long time without being made aware of Contraption Maker, for the creators of The Incredible Machine; this matters beyond nostalgia, as The Incredible Machine is one of the best pedagogical tools disguised as a game that I have ever come across, and Contraption Make is a worthy successor, improving it on many points such as the addition of rotation physics (want to do a cat flap? You can now.) or more digital logic elements for laser computing than you can shake a stick at.

And the last such discovery is for Two Point Hospital, in which you will find everything you liked from the original Theme Hospital; I haven’t played it yet, but if you’ve played the original and the trailer doesn’t convince you, I don’t know what will.

Copland 2020

Five years ago, I predicted that Apple would port iOS to the desktop, with a compatibility mode to run existing Mac OS X programs; we are now at the deadline I later put for this prediction, so did Apple announce such a thing at WWDC 2018?

Big No on WWDC 2018 Keynote stage, with Craig Federighi on the left

Now I think it is worth going into what did come to pass. Apple is definitely going to port UIKit, an important part of iOS, to the desktop for wide usage; and these last few years have seen the possibilities improve for distribution of iOS apps outside the iOS App Store or Apple review, though they remain limited.

But beyond that? I got it all wrong. The norms established by the current Mac application base will remain, with apps ported from the mobile world being only side additions, there will still be no encouragement for sandboxing except for the clout of the Mac App Store, pointing with pixel accuracy will remain the expectation, most of iOS will remain unported, etc. You are all free to point and laugh at me.

And I can’t help but think: what a missed opportunity.

For instance, in an attempt to revitalize the Mac App Store Apple announced expanded sandboxing entitlements, with developers on board pledging to put their apps on the store. Besides the fact some aspects of the store make me wary of it in general, I can’t help but note this sandboxing thing has been dragging on for years, such that it ought to have been handled like a transition in the first place; it might have been handled as such from a technical standpoint, but definitely not from a platform management standpoint (I mean, can you tell whether any given app you run is sandboxed? I can’t) even though that could be a differentiator for Apple in this era of generalized privacy violations. Oh, and that is assuming the announced apps will eventually manage to be sandboxed this time, and this is far from certain: I still remember back in 2012 the saga of apps that were worked on to be sandboxed, only for the developers to eventually have to give up…

I mean, Apple could have done it: the user base was in the right mindset (I did not see a lot of negative reactions when news of the unified desktop user experience got broken by the press a few months ago, which was in fact Marzipan, the initiative to run UIKit on the desktop), developers would obviously have been less enthusiastic but could be convinced with “Benefit from being one of the first native apps!” incentive, influential users could be convinced by selling it as a privacy improvement (remember: in Mac OS X unsandboxed apps can still read all the data of sandboxed apps), etc. But this year they explicitly prompted the question in order to answer it in the negative, meaning no such thing is going to happen in the next few years, and in fact on the contrary investing in alternative solutions like Marzipan. Sorry Apple, but I can’t help but think of Copland 2020.

Bruno Le Maire, Apple, and Google

A quickie because everyone’s mentioning it: France’s finance minister Bruno Le Maire announced he’d be suing Apple and Google for anticompetitive behavior with regard to people developing apps for their respective smartphone platforms. Here is the source (via).

The translation in the Bloomberg article (via) is more or less correct, with the exception that Bruno Le Maire only mentions “the data”, not “all their data”. So you’re not missing out on much by not being able to listen to him in the original French.

Now as to the contents of the announcement… I sure hope the actual suit is drawn from better information than what we’ve been given here, because while I’m on the record as deeming the current system of exclusive distribution through an app store (something which Google isn’t even guilty of) as being unsustainable in the long run, to have any hope of improving the situation through a suit Apple should be blamed for things it is actually doing. For instance, developers do not sell their wares to Apple (or Google) by any definition of that word, they do have to use a price grid but have full latitude to pick any spot in that grid, and Apple at least does not get that much data from apps.

Beyond that, I believe there could be a case here, but I see many more matters where Apple could be, shall we say, convinced to act better through public action from consumer protection agencies, antitrust agencies, or tax enforcement agencies. Why aren’t there announcements about those too? At any rate, I’m not going to breathlessly pin my hopes on that action until I get more, and more serious, information about it.

On the management of the performance of older iPhone models

When the coverage that Apple had admitted to capping performance of older iPhone models (when it detects their battery may no longer be able to withstand the necessary power draw) reached the (French) morning radio news I usually listen to, I knew Apple had a PR crisis of epic proportions on their hands.

Much has been written about this covert update to these iPhones, but the most important lesson here is the illustration that, once again, Apple completely and utterly fails at communication. If that PR crisis is not taken by the higher echelons as a wake-up call for Apple to take communication seriously from now on, what will?

Let us back up a bit, at the time Apple engineering was devising this solution for the (real) issue of some devices spontaneously resetting because the battery could not sustain the instantaneous power draw. We know it is the case that this was a project of some sort, in particular because the solution was rolled out for some models first, then other models, etc. Such a thing was obviously documented internally, because it is an important change of behavior that their own QA teams will notice when qualifying the software for release, also because it resolves a support issue, so obviously customer support was in the loop so as to provide feedback on which compromises are acceptable, etc. And yet, at the end of the day, when the fix is about to widely land in people’s phones, the system inside Apple is so completely stuck up on secrecy that not even an extremely expurgated version of this documentation makes it out the door? What is wrong with you people?

As a developer-adjacent person, I can see many developers being affected by this when they can’t understand why the performance of their product has regressed, but this pales in front of the perception by the general public of the whole affair… Indeed, many perceive (with or without justification, doesn’t matter) their computing products as slowing down over time, and given a lack of an easy-to-pin-down cause (I think it’s partly perception, and partly the compounded effect of multiple reasonable engineering tradeoffs), they are more than ready to embrace conspiracy theories about this. And now you’ve given them an argument that will feed conspiracy theories about this for decades! Even for issues completely unrelated to this performance capping (e.g. more pervasive performance pathologies). Stellar job here, Apple! Engineering-wise, I can’t fault the technical solution, however faced with someone affected by the issue, I can’t see how I or anyone, even an army of Apple geniuses, could credibly defend Apple. For all of us who trust on Apple to make reasonable compromises on our behalf (because we want to buy a computer/tablet/phone, not a computer/tablet/phone-tinkering hobby), and who told their interlocutors to place the same trust, you have damaged our credibility — and yes, it is something that should worry you Apple, because our credibility is part of your brand, ultimately.

On this particular issue, now that it has reached criticality, I think Apple made the right call in offering to replace batteries for $29 when they are degraded enough to trigger the capping, but that of course reeks of PR damage control, because that is (also) exactly what it is, so the bigger question of Apple’s capability to communicate remains open.

And I can’t close without saluting the work by the Geekbench crew to narrow down and confirm this effect. I’ve been critical of their product in the past, and I still am, but I have to recognize that it has allowed to bring this action from Apple to light, and I am thankful to them for this.

iOS 11 and its built-in App Store: weeks six and seven

Following the disruptive changes in iOS 11 and the fact we will have to use its redesigned App Store app going forward, I am diving headfirst and documenting my experience as it occurs on this blog. (previous)

Not much to report on for these two weeks, as I’ve been very busy covering the Saint-Malo comics festival for Fleen, then writing up these reports, then I had to compensate for lost time on the day job…

Though in a sense, it is an experience I can report on, as I’ve covered this festival, and in particular taken notes, exclusively using an iPhone (5S), an iPad Air 2, and Apple Wireless keyboard, and the setup worked very well. Half of the reports themselves were also typed up with the iPad. The notes were taken, appropriately enough, in the notes app, and reports typed up directly in the mail app (I used one or two additional apps, e.g. iBooks to keep offline access to the festival schedule). It is hard to say how much I benefitted from the new system functionalities, especially as they relate to multitasking, compared to what was already present in iOS 10, but on the other hand I feel they served me well, no regression.

  • I did have to relearn how to put two apps side by side, here notes and mail, but that was only a small learning bump.
  • The system generally does not allow pasting of raw text… which is an issue when composing email with data copied from many different sources. Get on that, Apple.
  • In the mail app the text editor would turn my quotes into French guillemets («»), and while I can explicitly specify straight quotes when using the virtual keyboard with a long tap, I have not found any way to do so with a physical keyboard… So I left them in; my editor had to contend with them when editing my piece for publication.

iOS 11 and its built-in App Store: weeks four and five

Following the disruptive changes in iOS 11 and the fact we will have to use its redesigned App Store app going forward, I am diving headfirst and documenting my experience as it occurs on this blog. (previousnext)

  • Last time, I forgot to mention that not only did the version number go directly from 11.0 to 11.0.2, but the latter was itself quickly superseded by version 11.0.3 (software update which I still do through my Mac for paranoia safety purposes, and regardless requires quite a bit of download). I wonder what happened there…
  • I use ellipses (…) quite often, and it took me some time to realize why I sometimes couldn’t find them on the iPad: they are gone from the French keyboard… I have to stick to the English one.
  • I haven’t managed to transfer old apps on older devices yet, but what I have done is uninstall a number of apps, especially ones that often get updated (Candy Crush, anyone?). This has resulted in a notable decrease of the number of apps I have to update at the end of the day, which is both a relief and a reduction of the download needs.
  • Speaking of which, in the storage preferences an attempt is made to provide the date of last use of the apps, but it does not take into account the use of the app through its extensions for instance (which includes a provided keyboard, a share sheet, etc.)

iOS 11 and its built-in App Store: weeks two and three

Following the disruptive changes in iOS 11 and the fact we will have to use its redesigned App Store app going forward, I am diving headfirst and documenting my experience as it occurs on this blog. (previousnext)

  • I think they went a liiiiiitle overboard with drag and drop. Case in point: I often make long presses in Mobile Safari to either preview the link or get the “title text” of an image (a very common practice in webcomics); but with iOS 11:

    1. I must wait even longer because the browser has to allow for the possibility for the gesture to be a drag and
    2. I must also move even less during that time because then the browser will interpret it as a drag.
  • When hitting an HTTP link in-app, it now always goes to the relevant app if there is one (instead of the somewhat random behavior from iOS 10), and no longer allows to go to Mobile Safari (through the top-right app+arrow symmetric of the top-left one that is still here and allows to return to the app in which the link was tapped). This is actually a regression for me, as when using Twitterrific I sometimes want to go to the Twitter app (to vote in a poll tweet for instance), but sometimes I do want to go to Mobile Safari, most often in order to open a new tab so that I can view the tweet later (e.g. it has a video and I don’t have my headphones on); in the latter case I now have to do copy, manually switch to Mobile Safari, new tab, and paste+visit. Another impact is that if the app refuses to load the content for some reason (e.g. Instagram refuses to do so if you are not logged in, and it may happen for multiple reasons, such as the fact multiple people may be using the iPad), there is no proposal to go to Mobile Safari, again needing a manual process (though in my case I solved this by uninstalling Instagram).
  • Speaking of which, Mobile Safari on iPad now no longer groups tabs in tab browsing view. I am neutral on the change.
  • In the Calendar app, you can no longer leave the event name empty. I use this in order to record instances where I miss a radio broadcast (e.g. the train was going through a tunnel while it was broadcasting): I just have to adjust the time and put it in a dedicated calendar, which makes recording this quick and easy (important so that I don’t end up forgetting). Now I also have to title those “Nothing”. Small but annoying.
  • On the iPad keyboard, when in symbol mode the key in place of the “shift” key is titled “#+=”, as in the iPhone… but contrary to the iPhone, these characters are not present in the keyboard you get after pressing that key. Uh?

iOS 11 and its built-in App Store: week one

Following the disruptive changes in iOS 11 and the fact we will have to use its redesigned App Store app going forward, I am diving headfirst and documenting my experience as it occurs on this blog (next).

  • Syncs are obviously much faster now. Beforehand the routine would be:

    • start the download of all app updates just before I leave home for work (…provided I remember)
    • once back home from work, sync the iPhone (~10 minutes)
    • then sync the iPad (~10 more minutes)
    • then trigger a Time Machine backup, to be sure to have the devices backups in there.

    Now it is more like:

    • once back home from work, sync the iPhone (~1 minute)
    • then sync the iPad (~1 more minute)
    • then trigger a Time Machine backup.
    • Meanwhile, tell my iPhone and iPad to download their app updates during the night.

    The undeniable advantage is that I can do these operations right after another without interruption, while with the long syncs I would have to leave to do something else while iTunes would do its thing to the iPhone or iPad… and often forget to launch the next operation until a few hours later.

  • I hate the new way the lock wall paper comes into view, because for split second you get the dreaded feeling the midtones are too dark. Indeed, if you’re like me you have bad memories of images being transferred from a Mac to a PC without gamma correction and the midtones appearing too dark as a result (same if you’ve worked on video or image editing, of you’ve prepped files for printing, etc.) I suppose I’ll get used to it at some point

  • The built-in QR code scanner works well. The implementation as a pseudo-notification is pretty nice to ensure you pick the right one when multiple are nearby:

    Interface of the built-in QR code scanner in iOS 11, inside the Camera app

    (this is from a sheet of paper I put to indicate the references of a few art pieces I put on my wall at work, in case my coworkers get curious)

  • I haven’t transferred the 32-bit only apps to the respective older devices (iPhone 3GS and iPad 2) yet. The hardest will be to transfer user data; I will probably need to start from a backup, merge with a backup of the old device (because some apps were already there), and reload that backup… fun stuff.

  • The new ability to uninstall apps is a godsend. Previously, in order to preserve game data (which took me a long time to obtain) weighing on the order of a few kilobytes (not even a megabyte), I was forced to keep 600 MB of data around like an anchor around the neck of this data. No more.

    The Chrono Trigger app shown as taking up 649.3 MB (app) and 41 kB (data)

    Note: you must have your Apple ID set up in the App Store app for it to appear (when I attempted to use it on iPad, I had just logged off from it for unrelated reasons, and was wondering why it would not appear as it would on the iPhone).

  • Animated GIFs put in the camera roll now do animate when you view them there. However, as a result you cannot edit them.

  • The Files app. This will be an ongoing section because I expect never to be done with it.

    Note the Files app does not provide a storage space by itself; rather, it is meant to unify the view between the different storage providers (iCloud Drive, Dropbox, Amazon Drive, etc.) on your device; I use Documents by Readdle. But it enables more than just this unified view.

    • A major change it permits is that you can now download unrecognized files, including blobs, from the browser to the Files app. Of course, that file has to go into one of the storage providers, but the action is different from the traditional “open copy in…”: the semantics are different, in the latter case for instance Documents by Readdle could decline to open the file which would prevent me from saving it there, while by going through the “Save in Files…” option in the share sheet, it is saved there without trouble.

    • However, there are limitations. In particular, Mobile Safari royally ignores the download attribute (rdar://problem/34745102/), including its value (blobs are saved with name “unknown.dms”), as well as the ”Content-Disposition: attachment” HTTP header (which forces the browser to download the resource, rdar://problem/34721730/): it always attempts to load the resource as a first step, which may be dangerous in some cases.

    • Also, all the file management issues we’ve been complaining for years for Apple to improve were dumped wholesale in the Files app, and in particular the presence of file typing extensions:

      In the immortal words of John Siracusa: “Would the real unknown please step forward”. Come on Apple!

    • However, if Mobile Safari is able to display or play the media, you seem to be out of luck: I have not seen any option to download it. Images you can save in the camera roll, but audio files for instance have no download option of any sort.

    Plenty more discoveries to follow, I am sure…

iOS app management removed from iTunes (a first reaction)

Perhaps a bit lost in the noise of last week’s announcements was the release of iTunes 12.7, which removes iOS app management (oh, and ringtones, too): you can no longer buy iOS apps on the desktop, or update them, or sync the ones you bought to your device except in an ad hoc way.

I admit I was taken by surprise. I heavily use these features: in principle, I do not download apps or app updates directly to my iPhone or iPad (there are exceptions): if I am at home with WiFi, I consider I might as well use my Mac, and elsewhere I’d rather not eat into my WAN bandwidth cap, battery life, etc. Plus, I indeed find iOS app browsing in the built-in (iOS) App Store app to be a substandard experience. Yes, some of us still don’t buy into the idea that the handheld device is necessarily self-sufficient; I mean I’d very much like to see you add freely distributed music (which as a result is not in the iTunes store) to your iPhone music library, or back up your iPhone to a non-Internet backup location, using solely the iPhone itself. As long as I can’t do that and have to sync, might as well use sync for everything (and honestly, I don’t mind sync per se).

And of course, speaking as a developer-adjacent person, I have to wonder what the impact is when potential customers who come across a link to an iOS app while browsing the web on their desktop… can no longer buy it there. There will be lost sales until Apple improves the situation (QR codes would be a start, for instance).

Now thinking about it more, I may be able to live without this feature. I don’t reorder apps on the iPhone screen from iTunes, app thinning means that even with two devices, my bandwidth use should even be lower (compared to downloading the “fat” app or app update as I do today), I haven’t tried to keep superseded version of apps just in case an update would ruin it, I will search and discover apps on the device if I have to (the main trouble in this scenario being, for me, the lack of free trials — that is not changing), I haven’t switched devices in a long time (though that might change in the next few months)…

So I’ll try it out. After one last sync, I will later today update iTunes and I will see if I miss anything. Maybe iOS 11 will help, maybe it won’t, maybe Apple will improve the experience (I wouldn’t hold my breath — iOS 11 features are already known, normally).

But as with yesterday’s post, my biggest worry is for historical preservation. What happens, in the long run, for apps that are no longer being sold in the iOS App Store? Will they only remain on devices where they were bought, with no chance of being able to transfer them on a different device? But I fear this is the last of Apple’s worries…