Alastair’s Place

Software development, Cocoa, Objective-C, life. Stuff like that.

On Security Monoculture

A pet hate of mine for some time has been the blanket assertion from those who like to identify themselves as “security professionals” that nobody should write their own cryptographic code. I’ve heard a number of individuals voicing this view and implying that all that is wrong in the world of computer security would be fixed if people would simply stop it.

This is, and has been, for some time, the conventional wisdom. It is wrong.

Why do I say this? Simple. The conventional wisdom implies that we should all be using the exact same code behind the scenes (this is often accompanied by claims of the superiority of Open Source implementations as they will be reviewed by many more people). For many people, and for many applications, this thinking leads to using OpenSSL, as it is “tried and tested”, and is Open Source so lots of people must have looked over the code and decided it was good, right? Well, let’s take a look at the huge list of vulnerabilities that have been found in that library, or the comments that the founder of OpenBSD, Theo de Raadt, made about it after deciding to fork it and create LibreSSL instead.

(Fine, you might say, use LibreSSL, or Botan, or Secure Transport, or CryptoAPI, or…; well, yes, that’s kind of my point. But I wouldn’t want to recommend that everyone should use LibreSSL, or Botan, or Secure Transport either. It’s much safer if there’s a mix of software performing this task.)

Heartbleed was only such a big problem because everyone was using the single implementation that contained that bug. Well, almost everyone; some software was using Apple’s Secure Transport, or Microsoft’s implementation (via CryptoAPI), or one of the various other implementations that are floating about. But the overwhelming majority uses OpenSSL, and as a result, a single vulnerability affected everyone, everywhere, simultaneously.

Another implication of this “thou shalt not implement crypto” view is that the set of implementations we presently have should be fixed. Maybe even some of them should go away. After all, nobody should be implementing crypto software (the only exception seems to be if the person quoting this rule knows your name, in which case you’re probably D.J. Bernstein or Bruce Schneier or some such). But that will make matters worse, not better. It will increase the reliance on OpenSSL and make the monoculture worse; and everyone switching wholesale to LibreSSL won’t help in that regard (it might be better in other respects, but that’s another matter). Indeed, it even implies that you shouldn’t be submitting any fixes to OpenSSL, because you can’t possibly be a suitable person to be tampering with cryptographic software.

Now, do I think you, dear reader, should immediately go out and roll your own RSA implementation? No, absolutely not. I am categorically not in favour of everyone implementing their own crypto (or, worse, rolling their own cryptographic algorithm). It isn’t something you can throw together in an afternoon, without carefully researching the subject first, and it certainly isn’t something you should be doing without adequate testing to make sure you haven’t slipped up. There are lots of gotchas in this area that you won’t appreciate unless you go and learn about it first. But what I don’t like about the conventional wisdom on the subject is that it has tended to discourage people who are competent to do so from writing additional implementations, and has created an atmosphere where you’re likely to be yelled at for merely suggesting that it might be a good idea for that to happen.

Code Coverage From the Command Line With Clang

Having searched the Internet several times to find out how to get coverage information out of clang, I ended up feeling rather confused. I’m sure I’m not the only one. The reason for the confusion is fairly simple; clang supports two different coverage tools, one of which uses a tool with a name that used to be used by the other one!

About half of the posts seem to indicate that the right way to get coverage information is to use the --coverage argument to clang:

1
2
3
4
5
6
$ clang --coverage -g -Wall testcov.c -o testcov
$ ls
testcov      testcov.c    testcov.dSYM testcov.gcno
$ ./testcov
$ ls
testcov      testcov.c    testcov.dSYM testcov.gcno testcov.gcda

This appears to produce (approximately) GCOV format data which can then be used with the gcov command, noting that this is really LLVM’s gcov, not GNU gcov, though it appears to be designed to be broadly compatible with the latter. Older versions of LLVM apparently used to call this tool llvm-cov rather than replacing gcov with it, but that name is now used for a newer, separate tool.

The rest of the posts, including some on the LLVM site, instead recommend using the -fprofile-instr-generate and -fcoverage-mapping options:

1
2
3
4
5
6
$ clang -fprofile-instr-generate -fcoverage-mapping -g -Wall testcov.c -o testcov
$ ls
testcov      testcov.c    testcov.dSYM
$ ./testcov
$ ls
default.profraw testcov         testcov.c       testcov.dSYM

Instead of outputting GCOV data, this generates a file default.profraw, which can be used with llvm-profdata and llvm-cov

The way to use this file is to do something like

1
2
$ llvm-profdata merge -o testcov.profdata default.profraw
$ llvm-cov show ./testcov -instr-profile=testcov.profdata testcov.c

In case you were wondering: you must pass the raw profile data through llvm-profdata. It isn’t in the format llvm-cov wants, and apparently the “merge” operation does more than just merging.

Also, you can change the name of the output file, either by setting the LLVM_PROFILE_FILE environment variable, or by compiling your code with -fprofile-instr-generate=<filename>. This is mentioned in the help output from the clang command, but doesn’t seem to be anywhere in the clang documentation itself.

In both cases, you need to pass the coverage options to the clang or clang++ driver when you are linking as well as when you are compiling. This will cause clang to link with any libraries required by the profiling system. You do not need to explicitly link with a profiling library when using clang.

One final remark: on Mac OS X, gcov will likely be in your path, but llvm-profdata and llvm-cov will not—instead, you can access them via Xcode’s xcrun tool.

Why NOT Have a Code of Conduct?

Having just seen a demand that WWDC adopt a formal Code of Conduct for its attendees this year (rdar://25791520 if you want to dupe it, though please give this post a read first), I thought I’d write a little to express my thoughts about the Code of Conduct phenomenon (in more than 140 characters, since that seems somehow inadequate).

Let me start by saying that it has always been the case that most conferences reserved the right to eject you if you were in some way disruptive. As private events, they’re within their rights to do so (at least in Common Law countries), and if they have the appropriate wording in their Terms & Conditions they may not even have to refund your money.

Let me also say that I am not in favour of allowing harrassment or other bad behaviour by conference attendees, and I realise that there will be situations (e.g. where there are children present) where the organisers might want to draw attention to the fact that attendees should keep to their best behaviour.

So what is this Code of Conduct thing about? Well, a fair overview is this FAQ by Ashe Dryden, and there’s an example of the kind of thing we’re talking about on this website. To save time, I’d recommend that you go and read those now, then come back if you’re still interested in what I have to say.

OK, you’re back. So why would anyone object to these things?

1. Necessity.

We’re grown-ups, right? We should all, by now, know how to behave around other people, and for those who don’t, we already have a set of rules that we’ve collectively agreed upon that cover the worst kinds of harassment and bad behaviour, namely the law, plus — as I already mentioned — most conferences already reserve the right to remove you if you’re being disruptive.

I accept, for what it’s worth, that some people might find an explicit set of rules reassuring. Others, me included, do not. Quite the opposite, in fact, for reasons I’ll elucidate below.

2. Natural Justice

It’s commonly asserted that a problem with leaving this up to the law is that the police “don’t have a great history of responding positively”, that complainants may not wish to involve the police and that as a result it might be better for conference organisers to deal with things themselves.

Except… conference organisers are not trained to deal with these types of situations. A lot of this is going to boil down to one person’s word against another, and it’s very easy to allow your own personal biases to determine your response. Police officers are trained not to do that (not always successfully, for sure, but they are at least trained); of course, that does sometimes make people unhappy when they complain to the police, because the police don’t seem to believe them — but that’s a misunderstanding. The function of the police is not to believe or to disbelieve, but to investigate, and where there is evidence, to bring it before a court for prosecution.

That courts of law require high standards of evidence — at least in Common Law countries — is undeniable, and that’s because we’ve collectively agreed that the principle should be that people are innocent until proven guilty.

This is particularly important in some of the areas we’re concerned with here, because of the reputational impact on people subject to allegations of sexism, racism or (worse) sexual assault, and the notion that the response to allegations of that nature might be decided by conference organisers on the basis of a low standard of evidence, without any right of appeal, really worries me.

I know it’s also asserted that “false accusations are… incredibly rare”. I’m happy to believe that. But there is a whole grey area of allegations that might seem true from a certain point of view that isn’t necessarily shared by all parties, and there are even situations where the accused and accusing parties simply don’t know themselves what happened.

3. Out-of-Venue Activities

Ashe Dryden asserts that a “code of conduct should apply to any event where your attendees may congregate”. This seems generally problematic.

Certainly there are situations where conference organisers might need to get involved; I accept that. But it seems hard to justify extending the Code of Conduct to all activities outside of those organised by the conference organisers.

So, for instance, if someone misbehaves in a bar right outside the conference venue, where there are a lot of conference attendees present, it is totally appropriate for conference organisers to have words with that person. Or, actually, for anyone present to have words with that person. But unless they have broken the law, or upset the bar owner, you won’t be able to ban them from hanging around in that bar, even if you kick them out of your conference. And, furthermore, to the extent that you feel the Code of Conduct may constrain their behaviour, it certainly won’t if you have invoked it to bar them from the rest of your conference.

Equally, it seems preposterous to argue that the Code of Conduct should extend to a shopping trip to a supermarket half way across town. Or to e.g. a group of attendees who decide to visit a strip club (not my cup of tea, but some people clearly enjoy that kind of thing, and it’s very likely effectively banned in the code of conduct you were thinking of using).

And then there are all kinds of questions about whether the Code of Conduct protects people who are not conference attendees at all, or indeed how it protects people who are conference attendees against those who are not (hint: it doesn’t).

4. Scope

What should be banned? http://confcodeofconduct.com suggests that “harassment includes offensive verbal comments related to… technology choices”! So you could, in theory, be evicted from a conference for making rude remarks about PHP (or, I suppose, for calling someone an idiot for using it). That seems a step too far, for sure.

In fact, while we’re about it, what constitutes an “offensive verbal comment”? Does it have to meet a reasonable person test? Would it be inappropriate to reproduce the cartoons of the Prophet Mohammed? In all circumstances? Are you sure? Does the whole of the community agree? Or if not, does everyone agree to compromise somehow?

And what exactly is “harassing photography”? Some people are very sensitive about having their photograph taken (even accidentally), and others much less so. Who decides? Is there a right of appeal? How many photographs does one have to take before it becomes harassment?

It’s also worth reflecting that quite a bit of that code of conduct would ban many well-respected and enjoyable comedy acts outright.

Again, please don’t misunderstand — I am all for conference staff taking someone aside and explaining that they’re upsetting someone, asking them to please be sensitive to that person’s concerns, and even if necessary warning them that they will be ejected if they continue with their behaviour. What I’m trying to tease out here is that there is a lot of subjective judgement involved, and attempting to codify this in a Code of Conduct is fraught with danger.

5. Legal Certainty

You might think that having a Code of Conduct would create some legal certainty for organisers when they do decide to act, but if they use the one at http://confcodeofconduct.com they could be in for a nasty shock. For instance, as it’s currently worded, it bans “offensive verbal comments related to… sexual images in public spaces”, rather than banning sexual images in public spaces as I’m sure its author intended. Granted, it says “harassment includes…”, so we can be certain that the definition is not exhaustive, but in cases where contracts are unclear, Common Law takes the view that they should not be interpreted in a way that favours the party that drafted them. My guess is that in court you’d find that they chose to use the legal definition of “harassment” (whatever that may be) and then added in anything in the “includes” list, in which case if you evicted someone for “following” and that person sued to recover their conference fees (and potentially travel and legal expenses), you might well find yourself out of luck and out of pocket.

Maybe that’s an argument for getting a lawyer to look over them, but IMO, it would have been much better to just put into the Terms and Conditions that the organisers reserve the right to eject attendees for behaviour that the organisers determine to be detrimental to other attendees or to the conference as a whole. I think you’d also want to make it clear what the procedure for doing that should be — who had the right to make the decision(s), whether there was an appeal process, under what circumstances attendees’ money might be refunded and so on. And on that subject, trying to keep hold of the entire conference fee whatever is probably a bad idea; the attendee’s credit card issuer is very likely to side with them, so if you’re going to try to keep hold of it you’ll only want to do so in cases where you have solid evidence of their misbehaviour.

6. Protecting “Unpopular” People

Some people may be unpopular, or may hold views that are unpopular. You can certainly discuss this with them in advance if you think it will be a problem, and ask them not to raise their unpopular views at your conference. If they aren’t relevant to the conference itself, they might even agree to that.

Anyway, there are two problems here; the first is that some people appear to claim that the mere expression of a view with which they strongly disagree is some form of harassment, in and of itself. Indeed, there have even been demands to ban certain people from certain conferences on the grounds that people are aware that (or think that) they hold certain views, even if they have promised not to express them at the conference.

The second problem is that unpopular people (or those with unpopular views) are far more likely to be the targets of false — or at least questionable — allegations. I don’t want to pick individual people as examples, so I’ll stick to generalising here: if a well-known feminist makes a joke about men, it’s quite unlikely that anyone will complain, and even if they do, quite unlikely that anyone will do anything about it. If, however, a similar joke about women was made by a man, I would expect there to be complaints, and I would expect that Something Would Be Done. (I’m not trying to be anti-feminist here; I’m just observing that, right now, at least in tech. circles, a fairly muscular form of feminism is popular, and making any remark that conflicts with or disagrees with that is not.)

It’s also worth reflecting that the first problem includes things like the views Roman Catholics or Muslims hold about homosexuality, which certainly for some people meet the definition of “offensive verbal comments related to sexual orientation”. While one might argue that people who hold those views should keep them to themselves for politeness’ sake (and indeed most do), if someone knows that they hold those kinds of views, they might be tempted to try to goad them into expressing them in order to trigger the Code of Conduct and get rid of those people from the conference.

The irony here is that the intention of advocates of Codes of Conduct is generally to protect minorities, but that in practice they may in some cases achieve the opposite.

7. Protecting the Expression of Unpopular Views (in some cases)

Sometimes it might actually be appropriate to prioritise freedom of speech over someone else’s right to not be offended. Sometimes it’s better to let people debate points of view that they may find challenging or even downright offensive.

I grant you that at most technology related conferences, this won’t be relevant, but I find Ashe Dryden’s assertion that this point can be addressed by stating that “free speech laws do not apply to harassment” overly simplistic, even leaving aside the obvious point that the United States Constitution, wonderful as it is, doesn’t actually apply over most of the surface of the Earth. It occasionally does all of us good to hear views we don’t like or agree with, even views we find offensive, if only because it makes us think.

(FWIW, I can imagine that this might become a problem if you wanted to have a conference talk or panel about gender politics in technology, which is something of a live issue at the moment; it’s very likely going to involve, one way or another, things that someone or other feels are “offensive verbal comments related to gender”. If you think not, imagine inviting e.g. Milo Yiannopoulos to debate with Brianna Wu, assuming you could get them to sit on the same stage.)

TL;DR

All of this is only my opinion, and hopefully I’ve explained above why I think this way.

  • Organisers certainly should have procedures to deal with poor behaviour by attendees, or with situations where one attendee is upsetting another somehow.

  • It would be wise to put these procedures in the Terms and Conditions.

  • It would be wise to train conference staff to follow these procedures (e.g. insisting that they report complaints up the chain to organisers until they reach someone you trust to deal with them sensibly).

  • Trying to codify what constitutes good or bad behaviour creates problems, and it’s probably better to use very general language in your Ts & Cs, instead of trying to write an explicit Code of Conduct.

  • If someone breaks the law, or is alleged to have done so, you really should consider letting the police deal with it, whatever your opinion of their effectiveness might be.

  • Attendees outside of your venue will be exposed to people who are not at your conference and are not subject to your Code of Conduct at all anyway (this potentially includes anyone you kick out for violating your Code of Conduct). As such, Codes of Conduct do not “protect” attendees. At best, if carefully drafted, they may protect conference organisers from future lawsuits.

  • Whether you have a Code of Conduct or not, you should consult a lawyer to avoid creating problems for yourself down the road.

  • There is nothing wrong with telling your attendees you expect them to behave themselves, drawing to their attention the fact that there are children present, telling them that you expect them not to stream pornography over the conference WiFi and so on. This is not the same as having a formal Code of Conduct.

Still TL;DR

Codes of Conduct mainly protect the conference organiser (and only if they are carefully worded); they don’t protect attendees. Defining what is and is not acceptable is hard, and boils down to subjective judgement anyway. Better to put procedures in place, stick them in your Ts & Cs, and train conference staff appropriately.

Symbolicating OS X Crash Logs

iOS developers have it easy; to symbolicate an iOS crash log, they can drop the log onto the Organiser window in Xcode, and — in theory at least — it will be symbolicated for them.

But on OS X, this doesn’t work. Moreover, the symbolicatecrash Perl script that iOS developers could use as an alternative doesn’t understand OS X crash logs and so will refuse to process them.

You could try using Peter Hosey’s Symbolicator package, but it’s a bit buggy — looking at the code, Peter has misunderstood the “slide”, and it also can’t cope with Xcode archives containing multiple dSYMs. I did contemplate fixing it and submitting a patch, but while I don’t want to be unkind to Peter, I think I’d end up rewriting rather too much of it in the process.

You could also try LLDB’s symbolicator, which you use like this:

$ lldb
(lldb) command script import lldb.macosx.crashlog
"crashlog" and "save_crashlog" command installed, use the "--help" option for detailed help
"malloc_info", "ptr_refs", "cstr_refs", and "objc_refs" commands have been installed, use the "--help" options on these commands for detailed help.
(lldb) crashlog /path/to/crash.log

This is actually really rather neat, or it would be if it worked. Unlike other symbolicators, it annotates the backtrace with your actual source code (and/or in some cases disassembly) so that you can see where the crash took place. Additionally, if you run it as above, within lldb itself, it will set up the memory map as if your program was loaded. Very cool.

You will note that I said if it worked. Because, out of the box, it does not. The first problem is that the version shipped by Apple relies on a script, dsymForUUID, that is not provided and whose behaviour is not documented anywhere. I wrote something that should be suitable and put it up on PyPI so you can install it with e.g.

$ sudo -H pip install dsymForUUID

(But wait… you might not need to.)

The second problem is that it’s also a bit broken. It chokes on some crash logs because they contain tab characters rather than spaces, and it also only loads the __TEXT segment in the correct place, which makes for a bit of fun if you need to poke around in one of the other segments.

Anyway, I filed a bug report today about all of this, with a patch attached to it that fixes these problems. I’ve also put a copy of the fixed crashlog.py file here so you can download and use it.

In addition to the usage shown on the lldb website, you can, in fact, invoke it directly from the Terminal prompt, e.g.

$ crashlog.py /path/to/crash.log

which is a very convenient way to use it in many cases. Likewise, if you want to use this version rather than the built-in one, you just need to make sure it’s in your PYTHONPATH, then you can do

$ lldb
(lldb) command script import crashlog

to use it in lldb.

The fixed version does not require dsymForUUID, and indeed it’s rather faster without it, but it can use a dsymForUUID script if you happen to have one (e.g. because you work at Apple). To use it with your custom dsymForUUID, you need to set the DSYMFORUUID environment variable to the full path of your script.

Update 2016-05-04

I found an interesting bug in the symbolicator; I’ve uploaded a new crashlog.py script that fixes it.

Update 2016-05-31

I’ve moved crashlog.py to Bitbucket, and added support for symbolicating the output of the sample command.

Apple Help in 2015

The last time I had to build a brand new help file was some time ago — maybe even ten years ago — and in the world of software, that’s an age.

For the past few months I’ve been working hard on a new release of iDefrag, version 5, and as part of this I’m rewriting the documentation. Rather than using hand-written HTML like I did before, I’ve chosen this time around to use a documentation generator, Sphinx. The advantages of this approach include:

  • Built-in support for indexing and cross-referencing.

  • The ability to write the documententation in plain text.

  • Keeps the presentation details separate from the content (via theming and templates).

  • Supports multiple output formats, not just HTML.

The current version of Sphinx doesn’t directly support building Apple Help Books, but I’ve submitted a pull request to fix that so hopefully by the time you read this you’ll be able to do

$ sphinx-quickstart

fill in some fields and then do

$ make applehelp

to generate a help book.

(If you do do that, you’ll want to edit your conf.py file quite a bit, and you probably don’t want to use the default theme either.)

Anyway, all of the Sphinx related stuff was fine, and worked as documented. Unlike Apple Help, which doesn’t. I spent an entire day struggling to make a help book that actually worked, and most of that is because of problems with the documentation.

Let’s start with the Info.plist. Apple gives this not particularly helpful table:

Key Exact or sample value
CFBundleDevelopmentRegion en_us
CFBundleIdentifier com.mycompany.surfwriter.help
CFBundleInfoDictionaryVersion 6.0
CFBundleName SurfWriter
CFBundlePackageType BNDL
CFBundleShortVersionString 1
CFBundleSignature hbwr
CFBundleVersion 1
HPDBookAccessPath SurfWriter.html
HPDBookIconPath shrd/SurfIcn.png
HPDBookIndexPath SurfWriter.helpindex
HPDBookKBProduct surfwriter1
HPDBookKBURL https://mycompany.com/kbsearch.py?p=’product’&q=’query’&l=’lang’
HPDBookRemoteURL https://help.mycompany.com/snowleopard/com.mycompany.surfwriter.help/r1
HPDBookTitle SurfWriter Help
HPDBookType 3
HPDBookTopicListCSSPath sty/topiclist.css
HPDBookTopicListTemplatePath sty/topiclist.xquery

There are two serious problems with the table above. The first is that some of it is wrong(!), and the second is that it doesn’t indicate which values are sample values and which are required.

Here’s what you actually need:

Key Value
CFBundleDevelopmentRegion en-us
CFBundleIdentifier your help bundle identifier
CFBundleInfoDictionaryVersion 6.0
CFBundlePackageType BNDL
CFBundleShortVersionString your short version string - e.g. 1.2.3 (108)
CFBundleSignature hbwr
CFBundleVersion your version - e.g. 108
HPDBookAccessPath _access.html (see below)
HPDBookIndexPath the name of your help index file
HPDBookTitle the title of your help file
HPDBookType 3

The first thing to note is that CFBundleDevelopmentRegion should have a hyphen, not an underscore. Apple’s utilities generate this properly, but the documentation is wrong.

The second thing to note is that in spite of the documentation implying that you can use your help bundle identifier to refer to your help bundle (which would, admittedly, make sense), you can’t. You need to use the HPDBookTitle value. Oh, and ignore any references to AppleTitle meta tags. You don’t need those.

The third thing relates to HPDBookAccessPath. The file referred to there must be a valid XHTML file. In particular, it cannot be an HTML5 document — that will simply not work, and the error messages you get on the system console are completely uninformative.

The best solution I’ve come up with for this particular problem, as I want to generate modern HTML output, is to make a file called _access.html and put the following in it:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>Title Goes Here</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta name="robots" content="noindex" />
    <meta http-equiv="refresh" content="0;url=index.html" />
  </head>
  <body>
  </body>
</html>

This means that both helpd and the help indexer (hiutil) are happy, and I can write my index page using modern HTML. Incidentally, Apple appears to be using a similar trick in the help for the current version of Mail. Obviously you can change the index.html in the above to whatever you need.

In your application bundle, you need to fill in the following keys

Key Value
CFBundleHelpBookFolder The path of your help book relative to Resources - e.g. SurfWriter.help
CFBundleHelpBookName The value from HPDBookTitle, above

Note that while the HPDBookTitle is displayed to the user, it can be localised using InfoPlist.strings. Note also that you absolutely cannot, contrary to what the documentation implies, give a bundle ID here. It just doesn’t work. You could however, if you wanted, write an InfoPlist.strings file like this:

HPDBookTitle = "SurfWriter Help"

then put the bundle ID in as the HPDBookTitle in the Info.plist.

Oh, and if you think you’re going to be able to double-click a help book to preview it, think again. That won’t work. Instead, you need either to use it from within your application, or you can put it in ~/Library/Documentation/Help (you might have to make that folder) and double-click it in there. Why? Because help files are indexed and you can only open them if they’re registered in the index.

One other thing that isn’t really documented at all is what exactly the HPDBookRemoteURL will do for you. There’s some handwaving about being able to offer remote content updates, but how the URL is used is skirted over. Well, if you do set HPDBookRemoteURL, Help Viewer will essentially expect it to point at a copy of the Resources folder of your bundle; so if you have HPDBookRemoteURL set to http://example.com/foo/bar/, then you’re going to get requests like http://example.com/foo/bar/en.lproj/index.html (and so on).

Useful update (Feb 29th 2016)

You may have noticed that Help Viewer has a button to toggle the table of contents in your help file. Matt Shepherd did a bit of work looking into this and it turns out that it’s controlled by a Javascript API — see Matt’s gist for more information.

January VAT Changes and the VAT Threshold

I’ve just spotted this petition, via a retweet from Dan Counsell, and as a member of HMRC’s Joint SME MOSS Working Group as well as the owner of a microbusiness I thought I’d make a couple of comments.

It isn’t particularly clear from the petition, but the problem being raised is that in order to register for the Mini One Stop Shop in the UK, you currently need to be registered for UK VAT. This is something that we have been talking to HMRC about, and I have the impression that HMRC is amenable, in principle, to allowing non-VAT-registered entities to use the Mini One Stop Shop system, though the details of that have not been worked out.

Note also that your sales here in the UK will continue to be subject to ordinary UK VAT, and will not be reported through MOSS, and even if your UK-only sales are below the UK VAT threshold, it’s likely that you have expenditure in the UK that involves an element of VAT, so you might want to consider a voluntary registration in any event, in order to reclaim your input tax.

(There is a related issue within the Mini One Stop Shop itself, in that there are no thresholds for amounts reported via MOSS. HMRC did try to negotiate a threshold, but other member states didn’t support the idea and it was dropped.)

It is also worth pointing out that the Mini One Stop Shop is optional. You don’t have to use it. The alternatives are:

  • Use a digital “marketplace” (e.g. Apple’s App Store, Google Play, Paddle). Marketplace operators, as of the 1st of January 2015, are required by law to deal with EU VAT for you. You will only need to deal with B2B transactions between you and the store operator.

  • Register for VAT in EU member states into which you are selling. This will mean filing multiple VAT returns and complying fully with (up to) 28 different sets of VAT legislation.

  • Use a distributor in EU member states you wish to sell into. The distributor is a business, so you only need worry about a B2B sale; B2C sales will be made by the distributor within the member state(s) in which it operates.

  • Stop selling to other EU member states.

For a lot of digital micro-businesses, the best approach is likely to be to use a digital marketplace. MOSS gets you a single return and a single payment; unlike using a marketplace or a distributor, it does not free you from the need to comply with up to 28 different sets of VAT rules, though it makes doing so considerably simpler in a number of ways.

As regards determining whether your sale is in the EU or not, with very few exceptions (mostly having to do with e.g. mobile network operators, where there is an obvious way to tell where the customer is) you need to keep two non-contradictory pieces of information that identify your customer’s location. These might include, for instance

  • Your customer’s billing address
  • The result of IP geolocation
  • Your customer’s telephone number

If those two pieces of information say your customer is outside the EU, then it doesn’t matter (from your perspective) if the customer was really stood in the middle of Brussels at the time; the rules say that you have done what is expected of you.

The Bash Bug

There are lots of scary headlines on the Internet today about a bug in the GNU Project’s Bourne Again Shell (aka Bash).

Apparently, Bash allows subshells to inherit exported function definitions, which it implements by passing environment variables with those functions’ names through to subshells, with the value of the variable containing the function definition. For instance

outer$ function hello {
> echo "Hello World"
> }
outer$ export -f hello
outer$ PS1="inner$ " /bin/bash
inner$ hello
Hello World
inner$ exit
outer$ export -nf hello

In this case, the outer shell has exported the function hello to the inner shell, by setting an environment variable hello to the string () { echo "Hello World"; }. We can test this:

outer$ export hello='() { echo "Hello World"; }'
outer$ PS1="inner$ " /bin/bash
inner$ hello
Hello World
inner$ exit
outer$ export -n hello

On its own, this feature is only harmful if a user can specify the name and content of an environment variable, and only then if some program is foolishly trying to run commands without specifying their full path. For example:

outer$ ls='() { echo "No way, Jose"; }' PS1="inner$ " /bin/bash
inner$ ls
No way, Jose
inner$ /bin/ls
foo.txt    bar.txt
inner$ exit

However, current versions of Bash contain a bug that causes Bash to execute trailing statements on environment variables of this form, so for example

outer$ naughty='() { :;}; echo "Oh dear, oh dear"' PS1="inner$ " /bin/bash
Oh dear, oh dear
inner$ exit

In the above example, the inner shell runs the echo command. It shouldn’t.

Now, this is potentially a major security hole, but only in certain circumstances, namely:

  1. If a user can set the value of an environment variable, and

  2. Where a program passes control to a Bash shell and passes that value through.

The two most common cases that you might find that allow remote exploitation of this bug are CGI scripts (the old fashioned kind, not FastCGI, and not anything run via Apache’s mod_php, mod_perl or mod_python) and OpenSSH if you were relying on the ForceCommand feature to provide restricted SSH access. sudo, fortunately, already strips out Bash exported functions (and has done since 2004), so is not affected.

Put another way, unless you have very old code running on your web servers, and unless you are doing something like running a public SSH server that allows restricted log-ins (e.g. to run Git or Subversion via SSH, but nothing else), the chances are that you aren’t vulnerable to remote exploits based on this. You should check, but you should not panic.

Twitter Is Not Private Chat

Let me say that again: Twitter is not private chat.

Why do I say this? Well, because it seems there are people out there who confuse Twitter with services like Glassboard, and think that people they don’t know shouldn’t respond to their tweets. Or maybe it’s just people who disagree with them; it’s unclear.

There are a few important facts that such people need to be made aware of:

  1. People who follow them may retweet their tweets. As a result they may very well be seen by people who do not follow them, who they do not know and who might disagree with whatever opinion they’ve expressed.

  2. By default, your tweets are public. That being the case, tweeting is like standing on a soap box at Hyde Park Corner, talking loudly to all who will listen. You don’t get to pick your audience.

  3. If you say something on Twitter (or indeed from a soap box at Hyde Park Corner), and someone who sees your tweet (or is listening to you) finds it interesting or controversial, they have every right to reply. Your “conversation” is not private in any way, shape or form; indeed, it is not actually a conversation.

If you don’t like the above facts, Twitter has a mode for you; set your account to “protected” tweet mode. At that point, you do get to screen your followers, who can’t retweet you.

Yes, there are downsides to protected tweet mode. If you don’t like the way Twitter works, and you don’t want to protect your tweets, post to a blog instead and turn comments off. Or use a private group chat system like Glassboard. Alternatively, you will simply have to live with it.

Finally, if you ask on Twitter why people are replying to you when you don’t want them to, and someone points out all of the above, there is absolutely no excuse for threatening or abusing them.

Code-points Are a Red Herring

Having just read Matt Galloway’s article about Swift from an Objective-C developer’s perspective, I have a few things to say, but the most important of them is really nothing to do with Swift, but rather has to do with a common misunderstanding.

Let me summarise my conclusion first, and then explain why I came to it a long time ago, and why it’s relevant to Swift.

If you are using Unicode strings, they should (look like) they are encoded in UTF-16.

“But code-points!”, I hear you cry.

Sure. If you use UTF-16, you can’t straightforwardly index into the string on a code-point basis. But why would you want to do that? The only justification I’ve ever heard is based around the notion that code-points somehow correspond to characters in a useful way. Which they don’t.

Now, someone is going to object that UTF-16 means that all their English language strings are twice as large as they need to be. But if you do what Apple did in Core Foundation and allow strings to be represented in ASCII (or more particularly in ISO Latin-1 or any subset thereof), converting to UTF-16 on the fly at the API level is trivial.

What about UTF-8? Why not use that? Well, if you stick to ASCII, UTF-8 is compact. If you include ISO Latin-1, UTF-8 is never larger than UTF-16. The problem comes with code-points that are inside the BMP, but have code-point values of 0x800 and above. Those code-points take three bytes to encode in UTF-8, but only two in UTF-16. For the most part this affects Oriental and Indic languages, though Eastern European languages and Greek are affected to some degree, as is mathematics and various shape and dingbat characters.

So, first off, UTF-8 is not necessarily any smaller than UTF-16.

Second, and this is an important one too, UTF-8 permits a variety of invalid encodings that can create security holes or cause other problems if not dealt with. For instance, you can encode NUL (code-point 0) in any of the following ways:

00
c0 80
e0 80 80
f0 80 80 80

Some older decoders may also accept

f8 80 80 80 80
fc 80 80 80 80 80

Officially, only the first encoding (00) is valid, but you as a developer need to check for and reject the other encodings. Additionally, any encoding of the code-points d800 through dfff is invalid and should be rejected — a lot of software fails to spot these and lets them through.

Finally, if you start in the middle of a UTF-8 string, you may need to move a variable number of bytes to find the character you’re in, and you can’t tell in advance how many that will be.

For UTF-16, the story is much simpler. Once you’ve settled on the byte order, you really only need to watch out for broken surrogate pairs (i.e. use of d800 through dfff that doesn’t comply with the rules). Otherwise, you’re in pretty much the same boat as you would be if you’d picked UCS-4, except that in the majority of cases you’re using 2 bytes per code-point, and at most you’re using 4, so you never use more than UCS-4 would to encode the same string.

If you have a pointer into a UTF-16 string, you may at most need to move one code unit back, and that only happens if the code unit you’re looking at is between dc00 and dfff. That’s a much simpler rule than the one for UTF-8.

I can hear someone at the back still going “but code-points…”. So let’s compare code-points with what the end user things of as characters and see how we get on, shall we?

Let’s start with some easy cases:

0 - U+0030
A - U+0041
e - U+0065

OK, they’re straightforward. How about

é - U+00E9

Seems OK, doesn’t it? But it could also be encoded

é - U+0065 U+0301

Someone is now muttering about how “you could deal with that with normalisation”. And they’re right. But you can’t deal with this with normalisation:

ē̦ - U+0065 U+0304 U+0326

because there isn’t a precomposed variant of that character.

“Yeah”, you say, “but nobody would ever need that”. Really? It’s a valid encoding, and someone somewhere probably would like to be able to use it. Nevertheless, to deal with that objection, consider this:

בְּ - U+05D1 U+05B0 U+05BC

That character is in use in Hebrew. And there are other examples, too:

कू - U+0915 U+0942
कष - U+0915 U+0937

The latter case is especially interesting, because whether you see a single glyph or two depends on the font and on the text renderer that your browser is using(!)

The fact is that code-points don’t buy you much. The end user is going to expect all of these examples to count as a single “character” (except, possibly for the last one, depending on how it’s displayed to them on screen). They are not interested in the underlying representation you have to deal with, and they will not accept that you have any right to define the meaning of the word “character” to mean “Unicode code-point”. The latter simply does not mean anything to a normal person.

Now, sadly, the word “character” has been misused so widely that the Unicode consortium came up with a new name for the-thing-that-end-users-might-regard-as-a-unit-of-text. They call these things grapheme clusters, and in general they consist of a sequence of code-points of essentially arbitrary length.

Note that the reason people think using code-points will help them is that they are under the impression that a code-point maps one-to-one with some kind of “character”. It does not. As a result, you already have to deal with the fact that one “character” does not take up one code unit, even if you chose to use the Unicode code-point itself as your code unit. So you might as well use UTF-16: it’s no more complicated for you to implement, and it’s never larger than UCS-4.

It’s worth pointing out at this point that this is the exact choice that the developers of ICU (the Unicode reference implementation) and Java (whose string implementation derives from the same place) made. It’s also the choice that was made in Objective-C and Core Foundation. And it’s the right choice. UTF-8 is more complicated to process and is not, actually, smaller for many languages. If you want compatibility with ASCII, you can always allow some strings to be Latin-1 underneath and expand them to UTF-16 on the fly. UCS-4 is always larger and actually no easier to process because of combining character sequences and other non-spacing code-points.

Why is this relevant to Swift? Because in Matt Galloway’s article, it says:

Another nugget of good news is there is now a builtin way to calculate the true length of a string.

Only what Matt Galloway means by this is that it can calculate the number of code-points, which is a figure that is almost completely useless for any practical purpose I can think of. The only time you might care about that is if you were converting to UCS-4 and wanted to allocate a buffer of the correct size.

Async in Swift

You may have seen this piece I wrote about implementing something like C#’s async/await in Swift. While that code did work, it suffers from a couple of problems relative to what’s available in C#. The first problem is that it only supports a single return type, Int, because of a problem with the current version of the Swift compiler.

The second problem is that you can’t use it from the main thread in a Cocoa or Cocoa Touch program, because await blocks.

As I mentioned previously on Twitter, to make it work really well involves some shennanigans with the stack. Anyway, I’m pleased to announce that I’ve been merrily hacking away and as a result you can download a small framework project that implements async/await from BitBucket.

I’m quite pleased with the syntax I’ve managed to construct for this as well; it looks almost as if it’s a native language feature:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
let task = async { () -> () in
  let fetch = async { (t: Task<NSData>) -> NSData in
    let req = NSURLRequest(URL: NSURL.URLWithString("http://www.google.com"))
    let queue = NSOperationQueue.mainQueue()
    var data = NSData!
    NSURLConnection.sendAsynchronousRequest(req,
                                            queue:queue,
      completionHandler:{ (r: NSURLResponse!, d: NSData!, error: NSError!) -> Void in
        data = d
        Async.wake(t)
      })
    Async.suspend()
    return data!
  }

  let data = await(fetch)
  let str = NSString(bytes: data.bytes, length: data.length,
                     encoding: NSUTF8StringEncoding)

  println(str)
}

Now, to date I haven’t actually tried it on iOS; I think it should work, but it’s possible that it will crash horribly. It is certainly working on OS X, though.

How does it work? Well, behind the scenes, when you use the async function, a new (very small) stack is created for your code to run in. The C code then uses _setjmp() and _longjmp() to switch between different contexts when necessary. If you want to cringe slightly now, be my guest :–)

Possible improvements when I get the time:

  • Reduce the cost of async invocation by caching async context stacks
  • Once Swift is fixed, remove the T[] hack that we’re using instead of declaring the result type in the Task<T> object as T?. The latter presently doesn’t work because of a compiler limitation.