One Very Important Thought

Now that the show is over and we have jointly exercised our constitutional rights, we would like to leave you with one very important thought, sometime in the future you may have the opportunity to serve as a juror in a censorship case or a so called obscenity case. It would be wise to remember that the same people who would stop you from listening to Boards Of Canada may be back next year to complain about a book or even a tv program. If you could be told what you can see or read then it follows that you could be told what to see or think. Defend your constitutionally protected rights, no one else will do it for you. Thank you.

From the end of Music Has the Right to Children by Boards of Canada.

Apparently this is sampled from the end of an adult film (SFW). “Jointly exercis[ing] our constitutional rights…” I like the euphemism. :)

Using object destructuring to implement named parameters in JavaScript

One of the neat things about JavaScript 1.7 and up is the support for array destructuring. It allows you to do things like

function foo() { return [5, 10]; }
let [a, b] = foo();

or even

function bar([a, b]) { return a + b; }
bar([5, 10]);

While the first example is obviously useful, the second one is somewhat less so. However, I didn’t know just how useful the general idea behind the second example is until this weekend, when asuth pointed out on IRC something I’d been previously unaware of: JavaScript supports object destructuring too. This means that you can do stuff such as

function baz() { return {a: 5, b: 10}; }
let {a: x, b: y} = baz();

Here, x and y will be 5 and 10 respectively at the end.

And yes, you can even write

function quux({a: x, b: y}) { return x + y; }
quux({a: 5, b: 10});

This looks like a great way to implement named parameters for your functions whenever you need them.

But wait – there’s more! There’s even a shorthand for when the property name is the same as the name of the variable you want to assign to. The last example could be rewritten as

function quux({a, b}) { return a + b; }
quux({a: 5, b: 10});

Depending on your use case, this might be an even better way to implement named parameters!

Note that the shorthand only works for destructuring assignments – you can’t use it to create (or “structure”) objects.

Long-lived JS objects without references considered harmful

I was having trouble getting the unit test for bug 466227 (make gloda index the headers of IMAP messages that aren’t offline) to work. The test involved the IMAP fake server which we have in the tree, and was failing consistently after fetching the headers of exactly six messages.

IMAP logging revealed that the connection was getting dropped right after the sixth message was completed, and that we were trying to reconnect to the fake server to execute the next command. Now, the fake server currently supports only one connection in its lifetime, so after getting the second connection attempt, it responded with an error message.

Clearly there was something unexpected that was causing the connection to drop, but we had no clue why. After a few false starts, I finally enabled logging on the socket transport service, and that revealed that nsSocketTransport::OnMsgInputClosed was getting called.

Setting a breakpoint there revealed nothing particularly significant (other than that an nsPipeInputStream was getting destructed) in the socket thread’s call stack, but something very interesting in the main thread’s stack:

We were in the middle of a JS garbage collection run – a forced one, no less. A DumpJSStack() on the main thread showed that this line in the gloda indexer was responsible. Sure enough, commenting out that line was enough to make the test pass.

bienvenu had the idea that one of the fake server’s internal objects was the one getting GCed, as we hold strong refs to any input or output streams that are given to the IMAP code.

After that, since I didn’t know of a way to uniquely identify the object getting GCed, it was a matter of brute force. Inside an nsPipe, the mStatus determines the status of a connection. So I set a breakpoint at nsPipe’s constructor, and then every time we hit the breakpoint, I ran a DumpJSStack(), noted down the JS caller along with the memory address on a sheet of paper, and set a data breakpoint for that pipe’s mStatus.* The third pipe was the one giving trouble, and that corresponded to this innocuous-looking line.

The fix was inordinately simple – just push the input stream into a long-lived array, thus making sure that we always hold a ref to it.

Huge thanks to bienvenu and asuth for all the help.

* Surely there’s a better way to do this (maybe using gdb instead)? i.e. run DumpJSStack() automatically, set a data breakpoint at mStatus, and log the DumpJSStack() output and the memory address somewhere?

(minor edit for clarity)

Support for Windows SDK versioning in mozilla-central

Mozilla uses the Windows SDK for official builds on Windows. Newer versions of the Windows SDK usually add support for features found in newer versions of Windows – for example, the Windows Vista SDK added support for parental controls, stock icons, and the new Vista method of file association registrations.

Unfortunately, we can’t expect everyone to upgrade to the Vista SDK, because you can’t install it on Windows 2000. Some people also build with GCC, which doesn’t support any of the Vista stuff either. Since Vista SDK headers can’t be included in the tree as-is, the solution for that was to add a new flag.

The Windows 7 SDK adds a few more features that we’d probably like to support, and it was clear adding SDK specific flags willy-nilly to configure.in is not a good solution in the long run. Patches to add a generic way to detect and enable SDK versions have finally landed on trunk.

What’s changed?

• The option --with-windows-version has been changed to mean the Windows version to target using this SDK (the default is 600, which is Windows Vista).
• configure.in will detect which SDK you’re building with, and will exit with an error if it’s too old.
• --disable-vista-sdk-requirements is now deprecated – you should use --with-windows-version=502 instead.
• Once the remaining patches on that bug land, you should be able to specify Windows SDK versions by saying #if MOZ_WINSDK_TARGETVER >= MOZ_NTDDI_X, where X specifies the minimum version of the SDK needed. Currently, X can be either LONGHORN, which should be used around code that needs the Vista SDK to build, or WIN7, for the Windows 7 SDK.
• Makefiles can’t be used for this any more.

What hasn’t?

• _WIN32_WINNT, WINVER and NTDDI_VERSION haven’t changed – you’ll still need to bump them up within a file if you need to.
• mozilla-1.9.1.

Seven things

I can has internet meme too? Thanks, jcranmer.

The rules.

1. Link back to your original tagger and list the rules in your post.
2. Share seven facts about yourself.
3. Tag some (none? :) ) people by leaving names and links to their blogs.
4. Let them know they’ve been tagged.

The things.

1. I’m a huge video gaming freak – I think I played my first game of Super Mario Bros. well before I first spoke. For some reason, for a long time I thought that the Koopas were ducks and not turtles.
2. My first web site had a lot of those “cool” Frontpage “DHTML” effects and animated gifs. My little mind at that time couldn’t even comprehend someone hating such an awesome thing. Oh, those were the days.
3. I’m not the kind of guy who picks fights, but had this knack to get involved in them. :(
4. I like to listen to a lot of classic rock, prog rock, blues rock, jazz-pop, and trip-hop, but I haven’t yet found a metal band that I like. (Opeth comes close – maybe I need to give them another proper listen.)
5. In my opinion, Arrested Development is probably the best TV comedy in existence. It isn’t really meant for TV, though; you need to be able to rewind or go back episodes to catch some of the subtler funnies in it.
6. I’ve been playing around with various Linux distros for close to eight years now. I remember installing Red Hat 7 for the first time around 2001. (I’m still not sure whether the CD I had was legal or not.) I even switched full time to Ubuntu for a while. All my attempts have ultimately led to frustration with something or the other, though, and I’ve been back on Windows for around a year and a half now.
7. I’m unable to parse images (like icons) as easily and quickly as text. However good the rest of OS X might or might not be, I don’t think I’ll ever be able to use it, as I have a lot of trouble with the OS X dock.

The taggees (sp?):

• Mark Banner
• Jeff Beckley (don’t think he has a blog, though)
• Josh Geenen, fellow Mozilla GSoC student
  
• Nirbheek Chauhan
• Naresh V
• Arun Chaganty
  

End of the Google Summer of Code

The Google Summer of Code 2008 coding phase ended a week ago, and final evaluations are currently on.

My GSoC project was about integrating Thunderbird with Windows Search, so that people can search for emails on Windows as conveniently and quickly as they search for documents and other files.

What's been done?

How much of what I initially set out to do has been completed?

• There's a patch up at bug 430614 that adds almost full fledged support for Windows Search integration on Windows Vista to Thunderbird. It seems like it should land for Thunderbird 3 beta 1.
• A lot of the backend code that sends notifications of messages or folders being added, copied, moved or deleted, has been fixed. These notifications are essential for the instantaneous indexing I hoped to achieve. Now, with the profile on an NTFS file system, and the Thunderbird and Windows indexes in their steady state, new messages should be indexed within seconds of arrival.
  Right now, the indexing code is the only core code that uses the notifications, but I understand that gloda will use it too for its incremental indexing. This should also be useful for extension developers looking to keep track of messages and folders.
  Fixing the notifications and writing tests for them did take a substantial amount of time, though.
  
• Spotlight integration on OS X, which was introduced in Thunderbird 2, has also received several fixes because of a lot of code being shared between the two.
• More code that has been rewritten includes GetMsgTextFromStream and the code to open Thunderbird from a search result.

What hasn't been done so far?

• A good UI within Thunderbird to manage indexing. The plan right now is to merge this with the default client dialog, presenting a unified "OS Integration" dialog. Let's see how it works out. If you have a suggestion, please let me know!
  
• Windows XP support. Windows Search for Windows XP lacks a "property handler" which can be used to parse MIME messages and present useful data about the headers in a UI, and also allow filtering results based on them. (This is built into Vista, and we make use of it. So, you can type in from:xyz in the search box and get results based on this.) An option is to write our own property handler.
• Proper support for indexing newsgroups. Specifically, notifications for deleted and copied messages.
  
• I guess that if support for indexing newsgroups lands, support for turning off search integration for particular accounts should also be present. Right now it's a global setting, but adding per-account support shouldn't be too hard. One thing to figure out is how messages moved between indexed and non-indexed accounts should be handled.
  Question: should the granularity be per-account or per-folder? Per-folder gives more control but has bigger problems than per-account.
• One concern that asuth had is that we currently send "message added" notifications for individual messages, as soon as the message is added to the database. If a new folder with thousands of messages is added, we'll have thousands of notifications, and those many XPConnect boundaries being crossed. One idea is to implement a token bucket system which batches multiple messages in one notification. This, like all asynchronous programming, has its own set of issues, of course: what if a message is added, but moved or deleted before its notification can be sent?
• Another aspect that deserves at least a brief look is the possibility of presenting results from Windows Search within the Thunderbird UI.
  

What have I learnt?

This was my first time participating in a large open source project, and I'm grateful to Google for the incentive to do so. My job definitely isn't over yet, and I'd like to continue to work on this until it gets into a very usable state.

(Apologies for any rambling below.)

• The final product might be completely different from the initial plan. The plan I'd put forward in my GSoC application just couldn't work.
  
• There are surprises everywhere. Every time you accomplish something and think that it's finally about to be in a usable state, you realize another part needs fixing. First it was the notification code. Right now my concern is with the UI, and clarkbw has some great ideas, one of which (the unified default client/search integration dialog) looks to be the way to go.
  Right now, though, apart from the UI, there don't seem to be any major, debilitating issues with the integration -- though I've hit upon a few edge cases, which seem to be caused when the account settings are absolutely mauled, where the search result handling breaks, and I'm looking for fixes to them.
• The old Windows developers had it easy. A separation of your program into administrative and non-administrative parts forces you to think carefully about what to add to each section, and about how to go about doing this with a minimum of hassle to the user. Right now we require one UAC prompt to enable integration and one to disable it, which looks like it'll be the minimum required.
  
• Wherever it can be used, automated unit testing works. It makes you be a bit more confident about the code than if you had relied on manual testing, and it protects against regressions that others or you yourself introduce. (I still go "doh" at that. :( ) Full credit to Standard8 for driving automated testing for mail/news.
  
• Tougher reviews are generally better. Old code that doesn't seem to follow any particular coding style is just plain harder to fix. Good, readable code is also easy to modify. Of course, there's the issue of patches being held up for too long.
  
• You can't assume any code that interacts with the OS platform, however old and well-tested, will work the same across platforms. Apple should -- no, must -- allow its OS to be virtualized, because debugging over IRC just isn't fun.
  

Thanks to...

(hopefully I'm not forgetting anyone)

Jeff Beckley, my GSoC mentor, for spending a ridiculous amount of time with me (I guess far more than I deserved), with guidance in both overall direction and specific implementation details.

David Bienvenu and everyone else at #maildev and #developers, for answering every potentially stupid question I had (though I hope the SNR wasn't too low :) ).

Leslie Hawthorn and everybody else in charge of GSoC at Google, and Gervase Markham, the Mozilla GSoC administrator, for making this possible.

... and Arun Raghavan, a GSoC 2007 veteran from my college, for holding a small lecture about GSoC (I think on March 13) that got me interested in it in the first place.

What a wonderful experience GSoC was!

The sheer pain of platform-specific code

So I submitted a patch to bug 436880, with some innocuous-looking tests. There were a couple of fixes needed to make the test work (and in general), but nothing major. All the tests passed on my Windows build like a charm.

All was well, until Standard8 tried out the test on his Mac and found that it failed with:

###!!! ASSERTION: You can't dereference a NULL nsCOMPtr with operator->()

Yes, a null pointer. After some initial shots in the dark (Time gaps before the next test? Line endings?), it was time for some debugging. It later turned out that when an instruction was given for multiple messages to be moved or copied, only the first one was actually being copied. As any further messages were not copied at all, we got a null pointer when we tried to retrieve them.

It was utterly surprising for me to see

1. such a problem being platform dependent.


2. such a basic operation having trouble -- except for the fact that this only happened in tests. (Don't worry Mac Thunderbird users -- your messages are safe ;) )


3. the problem happening only when multiple messages were moved or copied in one go.

I filed bug 439925 regarding this. One major problem was that I don't have a Mac to test on -- so all I could do was ask Mac users to run different tests with debugging information. (Thank you once again, Jeff and Standard8, for your valuable time.)

Finally, this Monday, David Bienvenu had some time to debug this on his Mac. He found that while copying messages from data files into a mail folder, the "folder" of the message object wasn't being set properly on OS X. (It was, on Windows). When multiple copies were done, we were sending the first message's source folder ourselves, but subsequent copies were using the "folder" set in the message object.

Now, why exactly wasn't the folder being set properly? David explains it best.

Lesson: Bad things can happen with non-normalized paths. (Non-normalized paths are paths that are something like foo/bar/baz/../../quux. Ideally, this path should have simply been foo/quux. Most of the time, both are equivalent. When they are not, as in this case on OS X, bad things happen.)

The fix was simple, just one line long.

All's (finally) well again with the world. It isn't. Again, this happens on OS X but not on Windows. Bah. Seems like I'll go through this all over again. :(