To anyone who has ever wanted a better hg manifest, or just a quick way to inspect a set of files controlled by Mercurial to see e.g. the last change revision: take a look at hglist.

Earlier today, Graham Lee linked on Twitter to a piece by Poul-Henning Kamp about the “tyranny of ASCII” in programming language syntax.

Kamp’s contention is that we should be free to use (for instance) “Dentistry symbol light down and horizontal with wave” (U+23C7, ‘⏇’ if your browser has it) as an identifier in a program if we so choose. Or, perhaps more reasonably, Ω₀.

It’s certainly an appealing idea, especially to anyone who has ever attempted to implement a mathematical algorithm, or even an otherwise non-mathematical algorithm that has come from an academic paper (which tend to use mathematical notation).

It is, perhaps, less than obvious what dangers await the unwary in this area; I think by now many people are familiar with the confusability of various glyphs, but perhaps it is not so obvious that e.g. ‘a’ and ‘а’ are in fact entirely different characters (the second is Cyrillic). Nor is it obvious to the uninitiated that ‘é’ differs in any way from ‘é’, or that care must be taken when comparing strings as a result, let alone more problematic character equivalences like ‘ß’ and “sz”/“ss”.

If we are going to propose allowing Unicode in identifiers, then, we need to specify:

• Which code points are and are not going to be allowed? e.g. Do we allow combining marks? Spacing modified letters? Private use characters?

• Do we allow bi-directional text? What about embedded bidi? If so, we can support Hebrew and Arabic, but matching identifiers is going to get complicated really quickly (the characters might be in either order, depending on the Unicode bidi rules)

• Do we allow identifiers to consist of characters from different scripts? For instance, is “аnd” a valid identifier? It isn’t the same as “and”…

• How are the compiler and linker going to determine a symbol table match? Does “æ” match “ae”? Does “ï” match “i”? What about “ß” and “ss”?

• What about the system linker (probably most important on systems with dynamic linking support)? Name mangling might be a solution, but we get exactly one chance to get that right before creating ABI compatibility problems. C++ didn’t do so well at that, as I recall.

Additionally, because different people have different fonts on their systems and not all code points necessarily have glyphs in all of (or even any of) those fonts, we perhaps need to think what will happen if a developer opens a source file on a machine that is lacking some glyphs that are needed to render an identifier. How is that displayed to them? Is it just a matter for their text-editor? Maybe so, but until text editors have a good way to deal with this situation, it’s potentially a bigger problem for the developer themselves.

Another concern is that, particularly in the case of Arabic, Indic and Far-Eastern languages, just typing “characters” on your keyboard can be quite an involved process. Certainly speakers of those languages may be familiar with the input methods they need to use, but that is not true generally, and if we permit identifiers containing such characters we risk fragmenting the developer community along natural language boundaries — people will write code that can only easily be used by a native user of their particular script. ASCII may be a poor subset of Unicode, but it’s very unlikely that any computer user anywhere on the planet is unable to type the majority of its characters. Unlike (say) allowing CJK characters in code, which means that anyone unfamiliar with the script will be relegated to using Cut and Paste (it isn’t even feasible in that particular case to attempt to locate the character by looking through a Unicode character palette — there are many thousands of different CJK characters).

Even if we restrict ourselves to (say) Latin characters plus accents, there’s still plenty of potential for confusion; just look at ‘ë’ and ‘e̎’ or ‘ç’, ‘c̡’, ‘c̦’.

For my money, then, allowing arbitrary Unicode identifiers is a mistake. I have nothing against the use of arbitrary Unicode in comments and string constants, but in the core syntax of a programming language more care is necessary in order to avoid creating problems.

Of course, some people retort that the issues I raise could be addressed by means of policies of individual projects. That’s true to a degree, though it still invites fragmentation of the developer community (which, I contend, is highly undesirable), and it doesn’t really address problems such as that of deciding on the equivalence of different characters/identifiers.

Personally, while I’m in favour, in principle, of expanding the set of characters we allow in programming languages, I think it needs to be done carefully and with considerable thought.

If you’re unlucky enough to be attempting to use Mac OS X Server to host users’ home directories, you’ll know that in recent versions the native AFP implementation has been getting worse and worse.

We’ve been seeing problems for a long time now, including files that you can’t delete, failures to save files, and even (occasionally) actual data corruption. Last Monday, everything came to a head; I tried to log-in, and after about a minute and a half, my machine was beachball city. Checking the log files, the culprit was quite clearly AFP (which had disconnected the AFP share in which my home folder exists, and was attempting to reconnect).

This ruined my day; I did nothing other than attempt to troubleshoot AFP. I tried reinstalling Mac OS X. No joy. Then I tried reinstalling it from scratch, having wiped the disk. Same problem. Next, I tried moving my preferences folder aside. That seemed to work, but today the beachballing was back. In fact, today, it was worse — it was affecting one of the other guys in the office too.

We’ve filed bug reports about most of these issues, though I wasn’t actually able to file the most recent bug because (ironically) Bug Reporter failed to respond every time I hit Submit.

Anyway, today, in order to let Ed, the other person seeing this problem, get on with his work, I had his home mounted over SMB for most of the day. Which was sort-of OK aside from the fact he couldn’t see files in his Downloads folder. At all. And a few other folders too. Turns out this is a known problem with SMB on Mac OS X.

So AFP doesn’t work. SMB doesn’t work. What other network filesystem can I use on Mac OS X (without messing about, and without going for non-POSIX-ish options like Coda or AFS)? Answer: that old UNIX staple, NFS.

On the server side, NFS was actually pretty easy to set-up; I just edited /etc/exports to add our users folder, then did sudo nfsd update.

The client side was a little more annoying.

We use Open Directory (basically LDAP, but with some sugar over the top to make it look more like Netinfo). OD has an attribute HomeDirectory, which looks like this:

<home_dir>
    <url>afp://server.example.com/Volumes/MyDisk/Users</url>
    <path>fred</path>
</home_dir>

Now, it turns out that Apple installs a neat tool called od_user_homes into /usr/libexec that can be used together with the default auto_master and auto_home configuration to automatically pick up the correct NFS share from Open Directory. All you need to do is ensure that the <path> part of the HomeDirectory is empty—i.e. this should work:

<home_dir>
    <url>nfs://server.example.com/Users/fred</url>
    <path></path>
</home_dir>

You also need to update the NFSHomeDirectory path to /home/fred, since we’re now using the auto_home system, but that’s straightforward.

However, and it’s a big however, if you do this, you’ll find that you can log-in just fine via ssh. What you can’t do, though, is log-in to a GUI session, and as usual with Mac OS X you get a cryptic message about not being able to log-in at the moment. Looking in the log files, we find

10/09/2012 16:41:07.039 authorizationhost[436]: ERROR | -[HomeDirMounter mountNetworkHomeWithURL:attributes:dirPath:username:] | Unknown URL attribute in user's home_loc (nfs://server.example.com/Users/fred)

GAH!

The problem is that the HomeDirMechanism bundle, which contains code to mount AFP and SMB-based home directories, doesn’t know what to do when it sees a URL starting nfs://, and, rather than doing nothing, it barfs, preventing users from logging in. HomeDirMechanism is the thing that was recently the subject of a security problem because it was logging passwords in plaintext. It is also the usual cause of the cryptic can’t-log-in-now problems. In fact, it’s just plain pants.

Since we’re using NFS now, we don’t need it!

We can just go to /etc/authorization and comment it out:

<!-- <string>HomeDirMechanism:login,privileged</string>
 <string>HomeDirMechanism:status</string> -->

Much better. Now we can log-in.

An unexpected and happy side-effect of switching to NFS has been improved performance. Vastly improved. When I did ls in my Source folder over AFP, the machine used to take several seconds to return with a directory listing. Now it’s instant.

There are, of course, some downsides to using NFS over AFP, but at least it’s reliable.

For those who don’t already know, on Saturday Jo and I got married at The Goodwood Hotel near Chichester. I’ll write more about this on another occasion, but for now I thought I’d put up a couple of photos that my father-in-law, Bob, posted on Facebook:

Update

Amanda Every, our photographer, has put a selection of shots up on her blog.

So, you ask, these ACLs… how can I read them from my own code? Well, you might think that the acl_get() family of functions were a good choice, and indeed, they do let you get at the ACL. Unfortunately, what isn’t mentioned is that, in addition to the usual UNIX uid/gid, files with ACLs may well have UUID-based owner and group information. There’s no acl family function to get that, but you’re going to need it if you want to understand what the ACL actually means.

If you look at Apple’s code (for instance, copyfile), you’ll see all kinds of undocumented functions, including such tasty looking morsels as statx_np() and chmodx_np(). Sadly they are not documented, and rely on further undocumented functions relating to the type filesec_t. These functions are visible (in <sys/stat.h> and <sys/fcntl.h>), but they are not documented.

At first sight it doesn’t seem that there is a documented way to access this data. But there is. getattrlist() allows you to retrieve ATTR_CMN_EXTENDED_SECURITY, ATTR_CMN_UUID and ATTR_CMN_GRPUUID (and setattrlist() lets you set them as well). Sadly it isn’t at all obvious how you go about getting from the thing getattrlist() returns to something you can manipulate as an ACL with documented APIs. The getattrlist() documentation unhelpfully notes that you get

A variable-length object (thus an attrreference structure) containing a kauth_filesec structure, of which only the ACL entry is used.

Well, you can see that structure in <sys/kauth.h>, but since it’s a kernel data structure, poking around inside it is naughty and you’ll probably get burned when Apple changes it.

How, then, can we get to an acl_t from this thing, without knowing what it is? Well, it turns out that the functions acl_copy_int() and acl_copy_int_native() expect their data in just this format. This is not documented — and it probably should be, given that Apple states in the kernel sources (in vfs_attrlist.c) that

/*
 * We have a kauth_acl_t but we will be returning a kauth_filesec_t.
 *
 * XXX This needs to change at some point; since the blob is opaque in
 * user-space this is OK.
 */

The other thing that it would be helpful to know is that the ACL data returned by ATTR_CMN_EXTENDED_SECURITY is in native endianness. So the function you need is acl_copy_int_native(), and not acl_copy_int(). Again, not documented.

So, to obtain this extended security information, we need to do

struct attrlist attrList;
struct {
  uint32_t        length;
  attribute_set_t returned_attrs;
  attrreference_t extended_security;
  guid_t          owner;
  guid_t          group;
  char            buffer[4096];
} attrs;
int ret;

memset (&attrList, 0, sizeof (attrList));
attrList.bitmapcount = ATTR_BIT_MAP_COUNT;
attrList.commonattr = (ATTR_CMN_RETURNED_ATTRS
                       | ATTR_CMN_EXTENDED_SECURITY
                       | ATTR_CMN_UUID
                       | ATTR_CMN_GRPUUID);

ret = getattrlist (path, &attrList, &attrs, sizeof (attrs), 0);

if (ret < 0) {
  // Handle errors
  ...
}

if (attrs.length > sizeof (attrs)) {
  // Need to use a bigger buffer in this case
  ...
}

acl_t acl = NULL;

if (attrs.returned_attrs.commonattr & ATTR_CMN_EXTENDED_SECURITY) {
  void *data = ((uint8_t *)&attrs.extended_security
                + attrs.extended_security.attr_dataoffset);
  acl = acl_copy_int_native (data);
}

at which point we have the ACL in acl, the owner’s UUID in attrs.owner (if attrs.returned_attrs.commonattr has ATTR_CMN_UUID set), and the group’s UUID in attrs.group (again, if ATTR_CMN_GRPUUID is set).

Likewise, if you want to set an ACL using setattrlist(), you’re somehow expected to know that you can create the blob of data you need with the acl_copy_ext_native() API.

Note that all of these functions and constants are documented, therefore in theory safe to use, though how you are supposed to use them together is not, and because Apple is using undocumented functions there is no good example showing how to use only documented APIs to find this information. Worse, the only example code I could find outside of the OS X sources, namely acl_api_fragment.c, makes use of the undocumented filesec and openx_np functions and is, therefore, a prime example of what not to do.

If you’re a Mac or iOS developer, you’ve probably heard of UTIs; if you come from some other platform, you probably haven’t, so I’ll start with a quick overview.

UTI stands for Uniform Type Identifier, and it’s used in a similar way to a MIME type—it’s a name for a particular kind of data—but unlike MIME types, UTIs are based on reverse DNS syntax strings (with two notable exceptions, which we’ll get to in a minute). If you want a new MIME type, you either have to register it with IETF, which is time-consuming and a source of uncertainty (they might not agree that you need one), or you can stick an x- prefix on it (but this might create clashes). If you want a UTI, however, all you need do is own an Internet domain; if you own example.com, you also own all of the UTI namespace starting com.example, so you could use com.example.MyDataType.

If you’re familiar with MIME types, you’ll also know that they’re split into two pieces, separated by a ‘/’ character. For instance, text/plain or image/jpeg. The part before the ‘/’ is supposed to tell you what kind of data you’re dealing with. This is useful, but it really doesn’t give you that much information; for instance, if you have a MIME type starting text/, you know it’s some kind of text, but you don’t really know how to decode it, or even if it’s compatible with US-ASCII.

UTIs, on the other hand, have a conformance hierarchy, so it’s possible to tell the system that your wonderful new data format, com.example.MyDataType, is actually a kind of video file. Moreover, you can be specific about it; maybe it’s actually a special kind of MPEG file. How does this work? Well, Apple has reserved all UTIs starting public., and has defined a large set of standard UTIs in that space. For instance, public.jpeg conforms to public.image, which in turn conforms to public.data, which, in turn, conforms to public.item. There’s actually more than one hierarchy (e.g. public.text conforms to both public.data and public.content), and Apple provides APIs that allow you to test any given UTI for conformance with any other UTI.

Additionally, every UTI has associated with it a set of tags. These are things like file extensions, MIME types and Mac OS OSType codes, and each class of tag is identified by—you guessed it—a UTI. So, the UTI for a file extension is public.filename-extension, for a MIME type it’s public.mime-type and for an OSType it’s com.apple.ostype. Again, APIs are provided that allow you to ask, given a UTI, e.g. what the file extension might be for that UTI.

Finally, Apple’s platform allows applications to declare their own UTIs by adding information to their Info.plist files, which makes them accessible to all applications on the system.

So what, you ask? Well here’s the really clever part; there’s an API that, given a tag class and a tag will find you the most appropriate UTI. For instance, if you give it public.filename-extension and jpeg, it will return public.jpeg. However, sometimes you’ll have a file that the system doesn’t recognize; you might know something about it (e.g. it’s file extension), but it isn’t in the system’s UTI database, and no application has mentioned it in their Info.plist. What happens now? Well, if, for instance, you ask for the UTI for a file you’ve found with the extension frob, you’ll get back a rather cryptic looking answer: dyn.age80q6xtqk.

The neat part about this odd-looking UTI is that the information you gave the system is still there. If you ask any Mac or iOS device for the file extension for dyn.age80q6xtqk, it will immediately tell you: frob. You could transmit this UTI across a network, and it would still tell you when you ask, that the extension is frob.

This runs slightly deeper, though. When you ask for a UTI, you can specify that it must conform to a particular type. Maybe you know that your frob file is really some kind of public.text? In that case, you might instead get the answer: dyn.ah62d4r34ge80q6xtqk. You might be able to guess that this new UTI also knows that it conforms to public.text, and again, any Mac or iOS device presented with this UTI will be able to tell you so.

Why is this feature useful? Well, web server administrators will have seen the unusual behaviour of various client systems when their server doesn’t know the correct MIME type for a file; some systems try to guess from the file extension, while others throw their hands up and treat the file as raw data (which is probably but not necessarily the default on any given web server). If the web used UTIs instead, the UTI would preserve all of the information the server did have, which would result in an improved experience in some cases.

What Apple doesn’t tell you is that these dynamic UTIs can actually hold more than just a single tag class. For example, the UTI

dyn.ah62d4r34gq81k3p2su1zuppgsm10esvvhzxhe55c

has a MIME type (text/X-frob), and a file extension (frob) as well as stipulating conformance to public.text. There’s no API on Mac OS X or iOS that will construct this UTI for you, mind… so how did I do it?

Well, the first thing you need to know is that all dynamic UTIs currently start with the string dyn.a. The first part, dyn. identifies the fact that it’s a dynamically generated UTI; the ‘a’ is, I’m guessing, a format identifier (so if you’re parsing these UTIs yourself, you must check that it’s an ‘a’ and not any other character; if it isn’t an ‘a’, you should indicate that you don’t understand the UTI).

Strip that off, and we have a funny looking string, which, it turns out, is encoded in a custom form of base-32 encoding using the encoding vector

abcdefghkmnpqrstuvwxyz0123456789

Let’s decode the dynamic UTI above and see what it contains; feeding the string

h62d4r34gq81k3p2su1zuppgsm10esvvhzxhe55c

through a custom base-32 decoder, we get

?0=7:3=text/X-frob:1=frob

Hmmm. Let’s look at another example; say we have a custom tag class com.example.SpecialType… let’s generate a UTI with that. Here it is:

dyn.ah62d4r34qr104pxftbu046dqqy1fg6dfqry0c5cytf2gntpgr710e2pw

Yikes! That’s a lot longer. What happened? Well, decoding it, we see

?0=7:com.example.SpecialType=foobar

Aha. So these are key-value pairs with the tag class and tag respectively. What about that funny ?0=7 on the front? Well, when I created this UTI I said it conformed to public.text. What if it conforms instead to com.example.Borked? Now I get

dyn.ah62d425try1gn8dbrz2g23mskm11e45fqu7gg55rf3w1u2prsb0gnpwxsbw0g4pbrvnhw6dfhzxg855cqf3a

which decodes to

?0=com.example.Borked:com.example.SpecialType=foobar

Hmmm. So ?0 means “the UTI we conform to”, right? So ‘7’ must mean public.text…

Well, it turns out that the system has a built-in list of UTIs that get abbreviated to the hexadecimal digits ‘0’ through ‘F’. They are:

0: UTTypeConformsTo
1: public.filename-extension
2: com.apple.ostype
3: public.mime-type
4: com.apple.nspboard-type
5: public.url-scheme
6: public.data
7: public.text
8: public.plain-text
9: public.utf16-plain-text
A: com.apple.traditional-mac-plain-text
B: public.image
C: public.video
D: public.audio
E: public.directory
F: public.folder

Now we can understand the string we saw earlier;

?0=7:3=text/X-frob:1=frob

expands to

?UTTypeConformsTo=public.text:public.mime-type=text/X-frob:public.filename-extension=frob

There are, as ever, a couple of niggles. The first is that tags might in general contain special characters (for instance, ‘=’ signs) that would mess us up. What does the UTI system do about these? It escapes them, that’s what. The set of characters that are escaped is:

, : = \ NUL

The ‘,’ is interesting; it turns out that each of the keys in the string can have more than one value associated with them. That is, it’s legal for the UTI to encode something like

?0=7,B:3=text/X-frob,image/X-frob:1=frob

resulting in something like

dyn.ah62d4r3qkk7dgtpyqz6hkp42fzxhe55cfvy042phqy1zuppgsm10esvvhzxhe55c

If you ask the system about this UTI, you’ll find that it conforms to both public.text and public.image, in spite of the fact that neither type conforms to the other. Unfortunately, Apple only provides a way to copy the preferred tag given a tag class and a UTI, so if you ask for its MIME type, you’ll only get text/X-frob (the first one, as per the documentation).

So, in summary:

• We’ve seen that UTIs can encapsulate any kind of type information, not just the obvious ones like file extensions. You can define your own type information, entirely orthogonal to the set Apple uses, if you like.

• We’ve seen that conformance information allows programs using UTIs to determine that (for instance) they know how to handle data with a UTI they’ve never even seen before because it conforms to a UTI they do understand.

• We’ve seen that it’s possible to generate dynamic UTIs that will remember the information with which they were created even when passed to another system.

• We’ve seen how these dynamic are generated, and the mechanism by which they hold the information given when they were created.

If you’re interested in reading Apple’s documentation, you might find their Uniform Type Identifiers Overview informative. Note that the format of the dynamic UTIs is undocumented. If you are going to rely on the information I presented above, make sure you check that they start dyn.a and do not assume that you understand any dynamic UTI that has a different format identifier.

Dear Adobe,

Signing up for Creative Cloud was not a pleasant experience. Here is why:

1. When I tried to go through your payment process, I was left with some flashing boxes and nothing else happened. This is apparently because I made the “error” of using Safari (Apple’s default web browser) to attempt to purchase from you.

   It is evident that you didn’t test your payment process in Safari properly (at least the last time you changed it) before you made it live. This is not acceptable.

2. You then had me download an installer for “Adobe Application Manager”. This is fine, but it failed, giving the error message

   

   Leaving aside for the moment the stray capital letter ‘p’ on the word “Please”, this message does not mean anything useful and it is not obvious what to do about it.

3. When I go to your support website and choose “Troubleshoot Creative Cloud Installation and Download”, I get this:

   

4. On further investigation, I was able to locate the installer that had been downloaded, which makes the following claim:

   

   Leaving aside the presentational inconsistency that the text in this window appears to be white rather than black, and the fact that in 2012 it is not reasonable (especially given the cost of your products) to merely throw your hands up when presented with a case-sensitive filesystem, the fact is that my filesystem is not case sensitive.

   Let me repeat for the hard of hearing:

   MY FILESYSTEM IS NOT CASE SENSITIVE

   Since you (Adobe) most likely won’t take my word for it (though I can’t imagine why), here’s a quick test in a Terminal window to demonstrate:

   

Now, having tried your software before, and having discovered in the past that the installers are totally broken, I was aware that the fact that I normally log in to the machine as a network user (albeit one with local administrator privileges — as a developer, that’s pretty much a necessity) was most likely going to cause your software to fail. Note: that is not any kind of excuse. It’s your shoddy work that makes it fail, not my choice to operate my computer system in an entirely reasonable, Apple-supported configuration.

I did eventually get your software to install, however:

• It is abundantly clear that you do not test your installers sufficiently well on Mac OS X. In particular, you need to make sure you address the following situations:

  1. When the user doing the install is not, themselves, an administrator.

  2. When the user doing the install is a network user whose home directory is on a fileserver somewhere.

  3. When the user doing the install has a case-sensitive home directory, but the rest of the filesystem is case-insensitive. THIS IS AN APPLE SUPPORTED CONFIGURATION and is quite common in set-ups where the fileserver is running some other flavour of Unix.

  There is no reason you should not be able to install successfully in all of these cases, even if you can’t be bothered to properly support case sensitive filesystems for your application bundles.

• It is also apparent that you did not properly test your purchase form with Safari, which is the dominant web browser on Mac OS X.

• You need to stop throwing your hands in the air when presented with a case-sensitive filesystem. It may not be supported for boot, but it is supported (at least) for peoples’ home directories and for other disks on the system. Some of these people might want your software installed in one of these other locations, and you should make sure that it works.

  If you are so utterly incompetent that you cannot work out that doing this is actually no more effort than making it work for case-insensitive systems, I have a library for you. Link against it, and all your case-sensitivity woes will go away. Though you may have some security bugs instead, if you don’t think too hard about it.

A result of all of this is that the purchase process was not smooth. As a developer, I was able to figure out what to do to get things to work; most normal users would not have been able to.

There is a story on the BBC website about a family who will be affected by the benefit cap, presumably to illustrate the argument of those who think that capping benefits is in some way wrong.

I have a couple of observations:

1. “Ray” is a software developer. Assuming he cannot get a job writing software, he must be a reasonably intelligent chap and should therefore be capable of getting himself all kinds of office work, never mind unskilled labour of one sort or another. Instead, apparently, he has been jobless since 2001.

2. The breakdown of their spending includes the following items, which, I submit, are luxuries that they have no right to expect the state (in the form of you, me, and everyone else) to provide for them. Namely:

   • Shows. This is listed, but not explained, under “Other”.

   • Entertainment. What Ray does on a Friday night is up to him, but if he doesn’t have the money, he doesn’t have the money. Maybe his friends would care to buy his beer for him, instead of expecting us to buy it?

   • Sky TV (perplexingly listed separately from “Entertainment”). This is justified with the wonderful “We get the Sky Movies package because we’re stuck in the house all week - otherwise we wouldn’t have any entertainment”.

     Of course (a) Ray does not have to be stuck in all week — he could get a job; (b) there is a perfectly good free TV and radio service; (c) there are always books and board games; and (d) if all else fails, there is the public library! I might add that public libraries often lend out films as well as books, just in case Ray has forgotten how to read while sitting on his arse.

   • Mobile phones. I don’t care that Ray says his teenagers will whinge at him if they don’t have them. They can’t afford because their dad can’t be bothered to find himself a job.

   • “24 cans of lager, 200 cigarettes and a large pouch of tobacco”. Really? 200 cigarettes costs at least £50, right there.

3. The amount that they will lose in benefits if this cap comes in is less than they are spending on tobacco and alcohol every week (I estimate this at the £20 “Entertainment”, plus £50 for 200 cigarettes, plus £15 for 24 cans of cheap lager and another £8 or so for the pouch of tobacco, which is £93).

The story ends with a quote from Ray: “I see eight people here having to choose between eating and heating.” Personally I see a lazy scrounger who can’t be bothered to go out and get himself a job. Any job. I don’t care if it’s as a software developer or as a damned toilet cleaner, quite frankly.

Earlier today, Stephen Fry linked to an article by Matthew Yglesias that posits that a little copyright infringment may actually be good for society.

The article makes the usual arguments about the over-estimation of economic loss to copyright holders, who, of necessity, talk about opportunity loss rather than concrete losses. Of course, in practice it’s impossible to come up with a definitive figure owing to the nature of copyright infringement—simply put, infringers don’t tell the copyright owner about their infringement (the only case where that can really happen is with software, and software that makes any kind of effort to do that usually upsets the privacy lobby). My perspective, as a copyright holder, is usually that even if we assume conservatively that only 10% of pirates would pay, given sufficient incentive, it would still represent a sizeable loss on any reasonable estimate of the number of pirated copies of our software.

However, the article does make a few more interesting claims. First, it claims (giving the example of a pirated TV show) that the loss, however large, from infringement is offset by the “$15 to $85 worth of enjoyment” that watching a pirated TV show would create. This, it seems to me, is a bogus argument. A car enthusiast may get £100,000 worth of enjoyment from driving his Porsche; that does not mean that stealing one from the dealer is no longer a loss to society. And it certainly does not make stealing a £50,000 Porsche a net gain to society of £50,000.

It also points out that the loss to the copyright holder is not necessarily an economic loss to society overall, as the infringer may use the money saved to (for example) visit a pizzeria. Again, this argument is suspicious; it seems to me that it would apply equally to mugging… for instance, if I am mugged for £100, which is then spent on burgers, I have lost £100, the burger joint and its suppliers have gained——and by Matthew Yglesias’ argument, society has not lost out overall.

Taken together, these arguments are even more suspect. Not only can I steal a Porsche and have my £100,000 worth of enjoyment, but the £50,000 I saved on buying it can now be spent as well! Society doesn’t lose out at all, and I can claim (as Yglesias does) that the entire £100,000 worth of enjoyment was a gain for society too. Win-win, right?

Yglesias then goes on to say that, because the BBC has yet to release the second series of Sherlock in the United States, he has been downloading it illegally over BitTorrent. Leaving aside for a moment my irritation that, as a U.K. license fee payer, Yglesias has just admitted stealing from me, it seems to me to be difficult to take him seriously when he talks about the pros and cons of copyright infringement if he is also indulging in it himself.

The article proceeds to claim that there’s a “considerable” benefit in forcing copyright holders to compete with “free-but-illegal downloads”, citing the existence of iTunes and Hulu as examples of legal options that he feels might not exist without pressure from piracy. Again, I find the argument rather thin; piracy is essentially identical, economically, to having a competitor who is engaging in dumping. I have yet to hear an economist argue that it would be good if goods and services were stolen and dumped in order to depress the market price. On the contrary, the usual view is that price dumping of any sort tends to force competitors out of the market, and in the case of piracy, the competitors are the people making all of the content that is being dumped.

As for whether or not there’s a problem on the consumer side—as distinct from commercial pirates—I think Yglesias’ analysis is facile. First, the current situation, where there is an excess of entertainment available to the consumer, is a hang-over from the previous situation in which making music and movies was a highly profitable business. There is still a lot of that money in the system and it will take time to drain away.

Second, there is a tendency to under-estimate the scale of the problem of consumer infringement. Talking to ordinary people (and even celebrities like Stephen Fry, actually, whose own income is dependent to some extent on copyright), will rapidly disabuse you of the notion that piracy is not a widespread thing. Many people I have spoken to boast openly about how clever they are to get things for free rather than paying for them. Ordinary people. Not computer whizz-kids, not stay-at-home living-in-mum’s-basement types. Yet everyone always assumes that “it’s just one copy”, “it’s just me”, “the movie/ music/software company is rich enough anyway” and so on. In a way, Yglesias has demonstrated that himself—he apparently feels that it’s socially acceptable enough to tell us that he’s illegally downloaded the BBC’s Sherlock.

When piracy was just a case of sharing something with your friends, it was less of an issue for copyright holders. Of course, many of them protested the illegality of doing so, but I think even they knew that it wasn’t hurting them that much overall. The problem is that the Internet has changed “sharing with your friends” to “sharing with anyone who cares to”; the scale has increased out of all proportion.

Finally, I think consumers fail to understand the motives of some of the players in this argument, and many of them end up—effectively—astroturfing on behalf of big corporations who are making a profit from others’ piracy. There is a reason that Google searches for The Pirate Bay still work. There is a reason that registrars providing WHOIS hiding services refuse to stop hiding the details of their customers even when they are egregiously infringing the rights of others. There is a reason that ISPs refuse to enforce their own Terms of Service. None of these things happen in the case of child pornography, but all of them happen for copyright infringement, even when it is blatant.

It is certainly the case that advertising and donations on dedicated piracy sites makes money for their operators. Money that should, rightly, be going to the people who produced the copyrighted content that they help to distribute, but which, right now, is going to line the pockets of the operators of the site, of their ISPs and registrars, of payment processors and of advertising networks. SOPA, above all else, appears to be an attempt to curtail that flow of money, and so it is hardly surprising that many of the companies involved are protesting about it, though their PR departments have obviously concluded that it’s far better for their respective images to frame it as a stance on the moral high ground of opposition to censorship rather than admitting their somewhat baser motives.

Everyone is probably aware that Wikipedia is blacking out its site today in protest at some new legislation proposed in the United States to discourage copyright infringement (namely SOPA and PIPA).

There are lots of breathless claims all over the Internet about the degree to which these bills will cause harm to the Internet, just as there were with DMCA before them. Indeed, people are talking about how any site might be taken down without notice, how payment providers and advertising networks might be forced to stop providing revenue streams and so on and so forth.

Most of these complaints are from people who have not bothered to read the full text of the bills, and are really just parroting what they have heard elsewhere. The result is that while they may be aware that SOPA could in principle be used to take down a site, they are unaware of the conditions attached to this, namely that:

• The owner or operator must be committing or facilitating the commission of criminal violations under sections 2318, 2319, 2319A, 2319B or 2320, or chapter 90 of title 18 USC.

• The site would be subject to seizure in the United States as a result of these violations if its owners or operators were located in the United States.

That is, in order for a site to be subject to take-down, it must already be breaking United States copyright law, and it would already be subject to take-down if its owners and/or operators were in U.S. territory. So, really, this part is just extending existing provisions in U.S. law so that they apply where the domain registrant is overseas. That seems fair enough, frankly, particularly as U.S. registrants might otherwise pretend to be overseas to escape the existing legislation.

Another thing that SOPA and PIPA do that is causing consternation is that they provide a mechanism for those whose rights are being infringed to notify payment processors and advertising networks that they must not process transactions for or make payments to the alleged infringer. This requires a notice similar to the ones specified by DMCA, and, just like DMCA, it is possible for the affected site to file a counter notice. And just like DMCA, if a counter notice is filed, it is the courts that must be used to decide what happens next.

They also create a limited immunity for anyone acting voluntarily to prevent copyright infringment; potential liability to their own customers has been used an excuse, historically, by registrars, payment processors and others, for continuing to allow their customers to egregiously infringe others’ rights even when their own Terms of Service explicitly ban such behaviour.

There have been all kinds of claims about the technical consequences of SOPA and PIPA, though most of these have been (as far as I can tell) baseless, since neither act makes any kind of stipulation about the technical measures that may or may not be used in its enforcement. I tend to think these are really a case of special pleading from a group of people who are making not inconsiderable sums of money from other peoples’ copyright infringement and/or are worried that enforcement might create additional costs for their businesses. These are not disinterested parties.

Anyway, regardless of your views on SOPA or PIPA, the blackout by Wikipedia is childish, affects countries other than the United States, whose citizens have no say whatsoever in whether or not the U.S. Congress or Senate pass their respective bills, and in addition has been done in a half-assed way.

For anyone who wishes to browse Wikipedia with Safari today, here’s a Safari extension that undoes the blackout.