Jonathon Rentzsch’s latest post about how his PowerBook/MacBook Pro’s keyboards don’t register some keypresses if both shift keys are held down was quite interesting, as was Dave Dribin’s Keyboard Matrix Help article that he linked to.

The phenomenon he’s noticed is quite normal and is due to compromises that are inevitably made when designing the layout for the keyboard’s PCBs and/or membranes. Even expensive keyswitch-based models like the Matias’ TactilePro on which I’m writing this article have their problems; mine, for instance, occasionally turns Caps-Lock on by itself when I type a sequence of keys. And if I hold down both Shift keys and try to type

QWERTYUIOP ASDFGHJKL ZXCVBNM

I get something like

WERYUIOP ASFGHKL "XCVBNM

Dave is right that you could theoretically fix most of these kinds of issues by adding diodes, though I think it’s a little unfair to suggest that their existence is entirely down to manufacturers “skimping” on a few diodes. For one thing, most modern keyboards are membrane-based, consisting of an insulating membrane with holes where they keys are, sandwiched between two membranes with conductive tracks printed on their respective surfaces. It’s quite hard to attach diodes to a rubbery membrane of that type (I don’t think it’s entirely impossible, since I’m pretty sure that there are companies out there offering the ability to attach surface mounted components to membrane circuits, but I don’t suppose it’s cheap). Two other issues that manufacturers have to consider are assembly costs (remember—you don’t have to place components on the matrix at all if there aren’t any, which saves an entire assembly step) and the physical geometry of the innards of the keyboard… obviously you can’t put diodes in spaces that are too small for them.

Also, the keyboard matrix designs in modern keyboards are usually pretty good

SQLite is quite a popular library, particularly on Mac OS X where it’s actually used to implement some system features.

Unfortunately, some of the filesystems on OS X (AFP, for instance) don’t implement advisory locking via the POSIX function fcntl(), which leads SQLite to report that the “database is locked” whenever you try to use it.

Now, SQLite is included with Mac OS X, and Apple have been kind enough to solve this problem for the version that they provide (3.1.3). Unfortunately, that version doesn’t support some of the syntax that gets generated by newer versions of SQLAlchemy, which I’ve been evaluating as part of a new design for my company’s website. So I needed to upgrade to a newer version to make it work, and since my home area is on an AFP share, I needed it to work on that.

The upshot is that I’ve ported Apple’s fix from 3.1.3 to the latest version of SQLite (3.3.6). Here’s a patch. To apply it, download the patch and the sqlite 3.3.6 sources (1.59MB). Then, in Terminal, do something like the following:

alastair$ tar xvzf ~/Desktop/sqlite-3.3.6.tar.gz
sqlite-3.3.6/
sqlite-3.3.6/art/
sqlite-3.3.6/art/tmp/
...
sqlite-3.3.6/publish.sh
alastair$ cd sqlite-3.3.6
alastair$ patch -p1 < ~/Desktop/sqlite-3.3.6-osx.patch

Then build it as normal. Note that if you run make test on an AFP volume, you’ll spot that one or two of the tests will fail. I think this is because of the differences in semantics between POSIX and AFP byte-range locks. If you run them from a local volume instead, they succeed as expected.

If you want the new build to match the version supplied by Apple, remember to configure it with —enable-threadsafe and —enable-tcl to turn on thread-safety and Tcl support.

Update 2011-10-14

I’ve removed this patch, as Apple has long since updated SQLite to a newer version.

John Gruber’s Mac OS X Tipping Point article remarks that

The myopic view of the classic Mac OS held by OS X era switchers, I suspect, stems from a thought process that runs like this: I wasn’t using a Mac in the late 90s, and I care about computers, and a bunch of my friends and colleagues feel the same way, so therefore no one who cares about computers was using a Mac in the late ’90s.

As a member of the group of “OS X era switchers”, I think this is manifestly unfair. It is a gross generalisation, and fails to take account of a number of factors affecting people like me. For instance, in the late ‘90s, I would have loved to have been using a Mac. As someone who previously used Atari ST machines through the late ’80s and early ’90s, the Mac was a case of a familiar (but slightly better) user interface, familiar (but slightly better) hardware and generally better (but more expensive) software. There are, of course, exceptions—anyone familiar with the latter days of the Atari ST range will remember remarks in the press about the number of Calamus DTP users decrying the lack of functionality in Quark Xpress, and many of the high-end music packages that today are run on the Mac and PC platforms originated on the ST platform.

However, as a student without a proper income, I wasn’t able to afford the new PowerPC Macs, and didn’t see much of a future for the admittedly attractive 68040-based machines that were still on sale. Nor were the clone machines particularly competitively priced (indeed, had I bought one of those, I would probably have regretted it, since they had the price point of Apple’s hardware but without any of the finesse).

So I was forced into the world of the PC and thereby Windows. And let me tell you, it is pretty horrible having to switch from an operating system that has a Mac-like UI with a mix of UNX and DOS-like underpinnings (in the latter days drifting ever closer to being UNX-like) to the frankly ropey UI of Microsoft Windows. Not that the Mac UI was perfect, but it was a lot better than Windows. I actually think that the Mac UI has got a bit worse, rather than better, with the introduction of OS X, but I’m hoping that Leopard will go some way towards resolving that.

You may ask why didn’t I then buy a Mac for ages after that? Well, although my income had increased somewhat, I had an investment in PC software at that point. The thing that originally tempted me into the world of the Mac was the fact that the titanium PowerBook was the best UN*X-based portable available. I wasn’t necessarily expecting to shift completely to the Mac, but when I found it so much less annoying to use than my PC, that’s what happened. And now, my business is based around the Mac too.

Ironically, perhaps, given that the ST was moving towards a model with a UN*X-like base and a Mac-like UI, modern Mac OS X has done exactly that! Though there are still a few things, to my mind, that Apple could have usefully learned from buying an old ST and looking at the software on it. For instance, being able to use windows that are not on the top, or the way that dragging a file from one window to another has the same meaning wherever the file is coming from or going to (on the ST, this would always copy the file… to move it instead, you had to hold a key; the Mac, like Windows, uses a ridiculous heuristic based on whether the file is on the same disk, forcing you—unnecessarily—to watch for UI cues that tell you what you’re going to do and modify your behaviour accordingly). Tear-off menus are quite handy too, as is a send-to-back button on the windows and the ability to resize from any side or corner. (I am aware, before anyone points out, that the original GEM didn’t have many of these features… but later GEM replacements, including Geneva, MagiC, nAES et al., certainly did/do).

It’s interesting to read Daniel Jalkut and Jonathan Rentzsch’s opinions of Apple’s new Dashboard auto-update mechanism.

What really annoys me about this particular issue is the sort of government-is-out-to-get-me, tinfoil-hat kind of hysteria that seems to affect both the press and many commentators on the Internet.

Let’s step back for a moment and consider.

Who is “out to get me”?

Well, it certainly isn’t the corporations.

Legitimate business does have an interest in sending marketing information to you, but it also realises that you won’t buy from people who persistently pester you. It is certainly true that the previous trade in personal details between marketing organisations has resulted in a backlash, notably including the creation of new laws in Europe and indeed other parts of the world. But, to re-iterate, legitimate business has nothing to gain from harming its customers.

However, the current darlings of popular culture, the P2P providers, along with their cousins in the cracking community (who write tools to help people steal software), are not legitimate. They will quite happily steal your information, since they’re already breaking the law —or, in the case of the P2P software developers, assisting you to break the law (however much they may disclaim such responsibility, there is not a P2P software developer alive who doesn’t know that the primary purpose of their products is to allow the general public to steal music, software and films without paying the copyright holders; their marketing on the Internet makes this pretty obvious, frankly).

The other group of people who are out to get you are those who steal personal information for profit, such as identity thieves, confidence tricksters, black-hat hackers and the like. They are very different from legitimate corporations.

What about the Sony “root-kit”?

Well what about it? What were Sony trying to achieve with their DRM-enabled CDs? The answer is that they were trying to stop the general public from stealing their artists’ music and distributing it over the P2P networks. Sure, what they did prevented some legitimate “fair use”, but they are entitled to be paid for material for which they own the copyright, and if the public are sticking their collective finger up to them in droves, they must protect the interests of their shareholders and their signed artists.

I don’t like DRM’d CDs particularly, and I don’t approve of surreptitiously installing software on users’ machines either, but, if they (probably rightly) concluded that what the RIAA is doing won’t work, and that the governments of the world aren’t really that interested in helping, what do you expect them to do? Tell their staff that they won’t be getting paid this month because that new album that just came out was distributed everywhere by P2P and as a result made a massive loss? Yeah, right, that’s fair.

If you, like me, think they did the wrong thing, that’s great. But most of us are lucky enough to live in democratic countries. If you accept that copyright isn’t working, tell your representative what you think should be done to fix it. Otherwise, someone (whether your government, or Sony, or some other party) will impose a solution that you don’t like, or (worse), the problem won’t get solved and there will be less music, less software and fewer films. Equally in a democracy, you should abide by the law; if you don’t like it, tell your representative, don’t go breaking it. The point of democracy is that we all get a vote and that rules are made on the basis of the opinion of the majority; sometimes you get your way, sometimes you don’t, but you have to accept that for the good of us all.

But I want to know what data is being sent and where

OK, so ask. Legitimate organisations have nothing to fear from telling you, and indeed in some parts of the world, they must tell you if they use some pieces of information about you.

How do I know that (for instance) Apple or Microsoft aren’t “out to get me”?

Legitimate businesses comply with the law, in all sorts of different ways. Even when things do go wrong, they tend to be problems relating to the finances of the business (as in the case of Enron, for instance) and are often driven by the fear of the Directors who, like habitual gamblers, find themselves in a mess but think they can trade their way out of it. Corporations very rarely wilfully break the law, though they sometimes do so accidentally. Wilful lawbreaking is certainly much less common amongst corporations than it is amongst individuals, partly because of the consequences that such activity can have in the long run.

Legitimate businesses are regularly audited, for all sorts of reasons. Shareholders may insist on audits to make sure the Directors are not up to no good. Directors may insist on audits to make sure that business processes are functioning as intended. Audits may be held for statutory reasons. Audits may also be held for reasons associated with particular certifications held by the business (e.g. ISO9000).

Legitimate businesses want to retain you as a customer. Unless you are especially obnoxious, a legitimate business will do its best to keep the relationship sweet. This is why when you buy a software package, you are typically offered free support, or discounts on upgrades to future versions. It isn’t just the business “being nice”—they want future business from you. It isn’t in their interests to steal your data and do something unpleasant with it; if they were found out, they’d stand to lose a lot more than they would ever gain (unlike those operating illegally or on the verge of illegality, who are already sticking two fingers up to the law and are more interested in taking your money quickly so they can get away with it).

But they’re watching my every move!

No, they aren’t. The vast majority if supposed intrusions on your privacy are actually cases where something is processed by an automated system and immediately thrown away. The reason for this is simple; storing the volumes of information that such things can collect is impractical for even the largest of organisations. Very often it’s uninteresting as well.

For instance, if you have the iTunes Mini Store turned on, it tells a computer at Apple what you’re listening to at the moment so that the Mini Store can suggest other things you might like (a useful service, actually, when you stop to think about it). Consider what information this provides to the machine(s) running the Mini Store service; obviously, there’s the name of the song you’re listening to… then there’s your IP address, the time, the date, the time zone, the name of the album, the name of the artist, and maybe other things too. Now consider that iTunes is reported to have over 18 million users. Assume that each of these listens to two songs a day and that they all have iTunes Mini Store enabled (as was originally the case when the feature was added). Then let’s suppose that the average size of the data sent to the mini store is 128 bytes (and that’s optimistic). That works out at over 1.5 terabytes of data per year. Doable, certainly, but expensive. Do you really think Apple want the information enough to set up and maintain a database that grows at that rate? I don’t.

Sure, at some point in the future, this kind of thing might be feasible, but let’s also think about what a legitimate businesses are really going to be interested in. They don’t care if you cross-dress at weekends, if you’re having an affair, if you spend your spare time surfing for porn, swinging or messing about with prostitutes. They don’t care if you’re racist or rude, if you’re a neo-con or a communist, if you’re gay or straight. Equally, they don’t care if you’re a vicar who spends his time nibbling cucumber sandwiches and listening to Mozart. They don’t care, to put it bluntly, what you do with your spare time. As long as you don’t steal their products or upset their staff, as far as businesses are concerned, what you do is between you and whatever god you believe in.

There are exceptions. Most businesses would be happy to report you if they think you’re breaking the law, e.g. if they catch you with pedophile images or plans to blow up the London underground. In some cases, the law actually requires them to do so, but then most people would want them to. And obviously, they want your business; so maybe if you buy products from their competitors rather than them, or if you’re obviously part of their target market, they’ll be interested—but that’s a very focussed sort of interest, and frankly one that we should all encourage because it’s competition that drives prices down and quality up.

But I don’t trust “Them”

Many corporations, particularly the larger ones, are publicly listed. That means you can buy shares in them. You can own them!

As a shareholder, you are responsible for appointing the Directors of the company. For instance, if Apple’s shareholders wanted, they could rid themselves of Steve Jobs and appoint someone else in his place. Obviously such actions are subject to a democratic vote based on the amount of money each shareholder is risking. Own more shares, and you get more votes. But that doesn’t stop shareholders from banding together.

Do you really think that a corporation is going to deliberately do something that would upset the people who own it? The people for whom the corporation is run, to make a profit? The people who can vote out the Directors at a drop of a hat. The people who can refuse to ratify the Directors’ pay, who could insist on endless audits, refuse to agree the accounts (which can, in extreme cases, result in the Directors being prosecuted and disqualified from holding such a post for some time) or even shut the company down completely?

Does that make sense to you?!

OK, some companies aren’t publicly listed, and in many publicly listed companies your individual voice is going to be small (though the opinions of e.g. pension funds, who represent many members of the general public can still be crucial to the running of the business). But most privately owned companies aspire to going public, with the huge increase in the value of their shares (and consequent increases in the wealth of their owners) that that implies. If you were going to do that, do you suppose that it makes sense to upset potential future shareholders?

Put simply, think of the effect on companies’ stock prices, and also the problems that it could create for them if they end up facing a shareholder revolt.

So why the fuss about privacy?

It is true, as I mentioned above, that some marketing organisations were selling customers details to other marketing organisations without mentioning it to or asking their customers first. This is definitely not the kind of thing that customers expect, and quite rightly there has been a backlash against this sort of thing, the result of which is that many countries have passed laws governing the trading of customers’ details.

But there’s still an enormous gulf between this and the sort of things that consumers should really be worried about (identity theft, blackmail, confidence tricksters, spammers and the like). Apple and Microsoft won’t be stealing your identity, blackmailing you, trying to scam you, or sending you dodgy e-mails about children and animals any time soon.

I‘ve read 1984. And I am not a number!

Great (no, really, it is). Just don’t get all paranoid about it. 1984 is fiction, and whilst some things from the book do have echoes in today’s society, you need to think very carefully before assigning the sinister motives of “Big Brother” to organisations simply because they seem to you to be large and powerful.

Nor should you blindly believe what the media has to say on the matter. Stories are often exaggerated or just plain mis-reported. It’s much more fun for the media to report sensational headlines, even when—as often turns out to be the case, if you look deeper—the reality is much more mundane.

So what should I do?

Chill out. Don’t foam at the mouth when you find a new automatic update mechanism that nobody told you about, or when you discover that some new feature happens to work via (shock horror) a server somewhere that has the potential to collate the data somehow. The real world is much less sinister than the make-believe world that some of the more rabid privacy advocates think they inhabit.

I’m too hot. And yes, I’m going to whinge about it to you :–)

It’s currently at least 25°C (air temperature) in this room (the thermometer I just fetched is still heading slowly upwards), and the Met Office people on the TV said this morning that it was going to be over 70% humidity. The air surrounding me feels like it’s probably gone over 90%, but then I’m a sweaty customer in this sort of heat so that’s hardly an objective assessment of the situation :–)

Of course, people in some parts of the world will look at this post and think “Hey, that’s a pleasantly cool day, what’s this guy talking about?”. Equally, in temperatures I feel more comfortable in, they might don ski-wear when I was still wandering about in jeans and a T-shirt. Funny how even peoples’ impressions of temperature are relative.

For some time now, it’s bothered me that a lot of people are being taught to write

int* a;

rather than

int *a;

The problem, as any experienced C programmer should be able to tell you, comes when you write

int* a, b;

Unfortunately, the interpretation implied by the spacing is that a and b are both pointers to integers. This is not, however, the case; instead, a is a pointer to an int, whilst b is an int.

The way the compiler reads this declaration mirrors the way that C programmers tend to write these kinds of statements:

int *a, b;

where it is obvious from the spacing that *a is an int (which implies that a is a pointer to an int) and b is an int.

The point is that the green part is the type, whilst the things you are declaring are the blue parts. The type of the resulting variable a is certainly pointer-to-int, but the declaration to the C compiler does not say that, it just says that a is an int, and to get the type of a, you have to read outwards from the variable name. This way of reading declarations works for all C and C++ declarations. The other way does not, forcing people to treat declarations like “int a” as a special case.

Writing things in such a way that a human being will read them differently to the compiler is just plain wrong. You can’t (as Bjarne Stroustrup tries to) justify it by saying that people shouldn’t write multiple variable declarations on a line—the C syntax (like that of C++) is perfectly clear and very easy to understand, provided you don’t try to space it as if it was e.g. Pascal.

It’s particularly telling that you can (if you like) write declarations using additional parentheses, like this:

int (*a);
int b, (*c), d;

though you can’t write

(int*) a;

(Well, not as a declaration, anyway…)

Of course, this syntax isn’t normally used by macho parenthesis-hating C programmers :–)

Anyway, consider this a plea, particularly to teachers of C and C++ programming: Please don’t teach your students to write ‘int* a’. It isn’t how the compiler reads it, and you will only make things more confusing for them.

Finally, for those people who like a puzzle, here’s an interesting declaration to decipher:

int (*(*fn)(const char *n))(int a, int b);

and you can award yourself extra brownie points for knowing—without using typedef or any pre-processor macros—how to write a function definition, a pointer to which could be assigned to a variable like the one above.

(Hint: think where you’re supposed to start reading the declaration to work out the type of fn.)

Having just criticised Apple for one thing, I’m going to add some praise for another. OS X 10.4.7 is a huge improvement for people like me whose home directories are mounted over AFP. I no longer get an enormous delay when I log in before I can do anything, which is great, and it seems (touch wood) to be more robust than it was before.

(Of course, just after I decided to write that, mdimportserver crashed…)

Some time ago now, in fact in November of 2003, I reported to Apple that it was possible to make the authentication dialog lie about which program was asking for authorisation to do something. This is filed as rdar://3486235, for any Apple employees watching.

Here’s an amusing demonstration:

(the program that did this was definitely not called “Steve Jobs”)

Very funny, but quite scary because it means it’s much too easy to trick an end-user into giving a potentially malicious program root privileges. Apple have been widely—and, to my mind, rather unfairly—lambasted for their attitude towards security holes, but in this case I’m sorry to report that the critics are quite correct. I’m sure they’ll fix this now I’ve published it on the Internet, but I really shouldn’t have had to do this; it should have been fixed back in 2003 when it was first reported.

The problem is that the dialog that appears gets the name of the calling program from argv[0], which as everyone knows is under the control of the calling program’s parent process. So, for instance, if you write something like this:

#include <unistd.h>

int main (void)
{
  execl ("/usr/libexec/authopen",
         "/Applications/Utilities/Disk Utility.app",
         "-stdoutpipe",
         "/dev/rdisk0",
         NULL);
  return 0;
}

which asks the special authopen program to give the calling process a file handle for /dev/rdisk0, what you’ll find is that it displays a dialog box like this

and in case you think that opening up the detail area will save you from getting caught out by a potentially malicious use of this bug, take a look at this:

Now, that definitely wasn’t Disk Utility asking for authorisation. It was the program whose source code is shown above. And if you authenticate, that program gets a file handle for /dev/rdisk0, which is clearly a huge security problem as it effectively gives it access to all of your data. Equally, it could use its own helper tool (rather than authopen) and simply ask for root privileges for whatever malicious purpose it had in mind. But this demonstrates the problem quite ably.

Ordinarily I don’t approve of people publishing security holes, because it puts end-users at risk, but this one makes it too easy to trick a user into giving away privileged access to his or her machine, and Apple still haven’t fixed it, over two years after it was reported. It’s also a sufficiently common issue that the comp.unix.programmer frequently asked questions document actually mentions that “it is possible to invoke programs with arbitrary values of argv[0]”, so the fact that you can set argv[0] to whatever you like is hardly an unknown feature, nor is it likely that the black hats don’t already know about this.

It was quite interesting reading Jonathan Rentzsch’s post about I/O errors and the impact that they might have on ordinary users.

As my company makes disk utility software, we see quite a lot of these kinds of errors, which users often report to us as “bugs”, telling us that they’ve checked the disk with DiskWarrior and it’s fine. Of course, DiskWarrior doesn’t actually check for bad blocks, which is where their mistake lies, so we end up having to explain that the problem is that our product stopped because it found something it couldn’t read, and that this was caused by a problem with their hard disk.

Unfortunately for us—though perhaps fortunately for most users—because of the way that modern disks “fix” them if you overwrite the bad area, together with the very large amounts of data held on disks that are infrequently accessed, the actual frequency is somewhat hidden and tends only to show up if you write applications like backup programs or defragmentation tools, both of which need to read a large proportion of the files on the disk.

Anyway, Jonathan, in his post, takes the hacker’s approach of trying to copy as much of the file as possible and for the bits that can’t be copied, ignore the errors and pretend the data was zero. This is fine for people like Jonathan, or like me, who understand about disks and computer programs and know the kinds of problems this might create. However, very few people in the general user population should really be trying to use tricks like overwriting blocks with zeroes or copying the file with options set to ignore the I/O error, since the consequences of those actions are not always wholly predictable.

To give an example, imagine that the document that has developed a bad block contains important financial information, and let’s also suppose that replacing the bad region with zeroes “fixes” it to the extent that the current version of whatever program wrote it is able to read it. Unfortunately, however, that does not mean that future versions will be able to read it, even if you use the “Save As…” option to save an entirely new copy of the data. So in five years’ time, when the IRS or (if, like me, you’re in the U.K.) HMRC inquire about the information contained therein, you might not be able to read it!

And in case you don’t think that problem is real, I’ve seen similar problems in the past that were caused by bugs in earlier versions that sometimes resulted in subtly corrupted documents, which crashed later versions of the same program when you try to load them.

What should you do?

Keep adequate backups.

That way, when this happens (and it is, I am afraid, a question of when rather than if), you can simply restore the affected file from a recent backup.

If you don’t have a back-up of the file, leave this kind of futzing with files to expert users. If you don’t know any expert users, then either:

1. Get the file so it works, and export it in plain text form (e.g. CSV, or a simple text file). Keep the plain text copy, just in case.
2. Contact a local computer club, or (if you really must) a data recovery firm.

I’ve found this great program that can recover data from bad blocks…

Such programs are only marginally useful, and at worst are just snake-oil, particularly if they claim to be able to use statistics about individual bits to recover their original values (that just won’t work with modern disks because of the fact that they use PRML and a Viterbi decoder; it might have worked, once upon a time, with MFM or GCR disks, and maybe even to an extent with non-PRML RLL disks, but the theory doesn’t work for disks using PRML).

Backups are a much better way to keep your data safe.

According to both the BBC and Wired News, Blue Security have shut down their anti-spam operation.

This is appalling because it amounts to an admission that the spammers attacking them are out of control and that unless law-enforcement action is taken they would be able to shut down the Internet completely.

Frankly, I for one would tolerate a few days of ‘Net downtime if it meant that the people responsible for sending spam went to jail. I don’t want tens of e-mails a day offering herbal Viagra or “teens with animals” or whatever other perverted trash they seem to want to send. Nor do I want endless attempts to fool me into giving away my bank details, my PayPal account details, or any of the other scams or cons that these guys send all of us. I just want to use the ‘Net in peace.