BuffaloFeeds » Friends

Friends (100 unread)

Comments for Josh the Outspoken

Comment on Refcounting immutable cyclic graphs by Josh

Posted: January 25th, 2012, 9:15pm MST by Josh

When a node is first created it is mutable (edges can be created or changed), but it does not own refs on nodes it points to, so the client of my library is responsible for keeping nodes alive through this first stage. When they client is ready they can “finalize” a set of nodes which does the SCC search and guarantees, from then on, that all reachable nodes are always kept alive.
Comment on Refcounting immutable cyclic graphs by Brian Slesinsky

Posted: January 25th, 2012, 8:33pm MST by Brian Slesinsky

If each node is immutable once initialized, how do you build a cycle to begin with? Do you build nodes in batches?
Comment on Refcounting immutable cyclic graphs by Josh

Posted: January 25th, 2012, 9:28am MST by Josh

Because my objects are immutable once initialized, the graph is immutable also and I can find SCC at initialization time, not when a refcount is decremented to 0. I do have to search, but I can do the search only once per graph.

This scheme wouldn’t work in most situations where GC or refcounting is used, but I have the luxury of an immutable object graph.
Comment on Refcounting immutable cyclic graphs by Dave

Posted: January 25th, 2012, 8:50am MST by Dave

How do you avoid the expense of frequently running the O(V+E) SCC-finding algorithm whenever an object’s reference count is decremented to 0? I can imagine coloring each object you visit as you go along in a normal refcounting approach, then when you detect an object you’ve already visited (b/c it’s been colored already) switching to the SCC approach. But this algorithm will probably be pretty abusive to the cache for large numbers of objects.

Or is it that in upb you (the programmer) naturally know which objects are SCCs because of some nice structural property of the code, so that you don’t need to “discover” where they are with some sort of search? Perhaps that’s what you mean by “if the defs are immutable”…

2 Star

Refcounting immutable cyclic graphs

Posted: January 21st, 2012, 12:07pm MST by Josh

Cycles are a thorn in the side of refcounting. But yesterday I discovered (or perhaps rediscovered) a simple, efficient scheme for refcounting cyclic graphs if the defs are immutable: find strongly-connected components and make all nodes in each SCC share a refcount. Beautiful.

This relies on the graph theory result that if each SCC in a graph is replaced with a single node, the resulting graph forms a directed acyclic graph.

I’m planning to use this scheme to refcount defs in upb (garbage-collection is a non-starter because I don’t want to have to track a global list of roots or force the client to periodically call a GC function).

Planet Matt

Bookmarks: Physicists Seek To Lose The Lecture As Teaching Tool

Posted: January 2nd, 2012, 2:59am MST

"'Students have to be active in developing their knowledge,' he says. 'They can't passively assimilate it.' This is something many people have known intuitively for a long time — the physicists just came up with the hard data. Their work, along with research by cognitive scientists, provides a compelling case against lecturing"
Journal: Pre School

Posted: January 2nd, 2012, 2:25am MST

As we start to think about school choice, I find it helpful to remind myself that (a) decisions we make now can still be changed later, and (b) real learning is not confined to or limited by the classroom.
Bookmarks: What If Middle-Class Jobs Disappear?

Posted: January 2nd, 2012, 2:12am MST

"These trends serve to limit the availability of well-defined jobs. If a job can be characterized by a precise set of instructions, then that job is a candidate to be automated or outsourced to modestly educated workers in developing countries."
Bookmarks: something went wrong

Posted: December 9th, 2011, 1:33am MST

Bookmarks: something went wrong
Bookmarks: Regular Expression Matching Can Be Simple And Fast

Posted: December 7th, 2011, 1:59pm MST

Bookmarks: Regular Expression Matching Can Be Simple And Fast

Comments for Josh the Outspoken

Comment on Wanted: a mailing list reader website by Pierre Riteau

Posted: November 16th, 2011, 3:12am MST by Pierre Riteau

I have also been looking for a while for such a gem â€¦ Did you discover any new project since you wrote this post?

Planet Matt

Bookmarks: The Kid Should See This.

Posted: November 15th, 2011, 10:14pm MST

"There's just so much science, nature, music, arts, technology, storytelling and assorted good stuff out there that my kids (and maybe your kids) haven't seen. It's most likely not stuff that was made for them... But we don't underestimate kids around here."
Bookmarks: MAKE | A Curriculum of Toys

Posted: November 11th, 2011, 4:09pm MST

"What are the fundamental things kids should know to help them understand and enjoy the complex physical world we live in, to modify or repair it in the future, to succeed as adults? How do we enable kids to be masters of their destiny? Can we do it with nothing but good toys and experiences?"
Journal: Some people say the sky is just the sky

Posted: November 6th, 2011, 11:42pm MST

Eleanor enjoys seeing the moon (a rare treat for a girl in cloudy Seattle, with an early bedtime to boot), so when I glimpsed a third-quarter moon through the skylight last night, I pointed it out to her, then pointed our telescope through the skylight for a better view. I had trouble explaining what craters were, so I grabbed a nearby tablet (since I started doing mobile development, they are lying around everywhere) and showed her some pictures from the lunar surface. She was disappointed the mountains weren't like the ones in Wallace and Gromit.

Next she wanted to see stars, so we went out on the back deck with a warm blanket. The moon and the city lights and the house blocked out a lot, but we did see a number of stars, plus Jupiter rising in the east. (Seeing Jupiter's Galilean moons through the scope was especially interesting to me because I'm in the middle of Galileo's Dream by Kim Stanley Robinson, which is set partly during the life of Galileo Galilei and partly on the moons themselves.) We stayed up past Eleanor's bed time, and I tried to answer her questions about planets and moons and stars and scientists. We used Google Sky Map to identify some of the things we'd seen outside.

Being around a five-year-old makes me remember how intense feelings and experiences were at that age. It's a lot of pressure for a parent, because every offer you make, or wish that you fulfill or deny, can lead to either thrills of pleasure or depths of disappointment. I don't have the energy to keep up with even half of what Eleanor wants to do, so I just work at finding enough I can manage. Yesterday she got to spend several hours playing with her best friend from last year's preschool class, which was perfect, Those two girls could keep up with each other so much better than I could hope to. Socializing is also hard work for Eleanor, though, and today she didn't seem to mind having a boring day at home.
Journal: Rewrite

Posted: November 6th, 2011, 10:36pm MST

At work we're in the process of rewriting Firefox for Android to replace most of our JavaScript/XUL front-end code with new code using Android's Java frameworks. This is looking like a very good move technically, but on a personal level it sort of cast me adrift. I've been working on the XUL front-end code for almost two years, and suddenly everything I've done or was about to do is living in a codebase that's soon to be abandoned.

Most of the team has jumped straight into the new front-end code, but I've had trouble doing that, partly because I had some loose ends to wrap up in the old code so we can ship the next few updates, and also because I was tired out from our last big project and didn't have the energy to jump right into another one. So I spent a couple weeks doing simple janitorial work like bug-fixing and sheriffing. This gave me some extra mental energy for my free-time projects like the AI class, and learning enough LLVM to contribute some patches to the Rust programming language.

I'm glad that I've learned to recognize swings in my productivity cycle. Instead of denial and procrastination during the low-motivation periods, now I try to accept them and use them to regroup. I think my anti-burnout strategy worked this time, since I now have some ideas of new projects I'm excited to try in the new codebase. If I'm lucky, that means I'm back on the upward swing of a new cycle.
Bookmarks: The Architecture of Open Source Applications: LLVM

Posted: November 3rd, 2011, 10:35pm MDT

Very nice introduction to what LLVM does and how it works.
Bookmarks: Lauren Ipsum

Posted: October 26th, 2011, 10:44pm MDT

A children's book about computer science.
Bookmarks: Rich Hickey: Simple Made Easy

Posted: October 24th, 2011, 10:10pm MDT

Rich Hickey explains why "simple" is important and "easy" is not; what the difference is; and the ways he has learned to make software simple.
Weblog: Android app permissions and Firefox Beta

Posted: October 11th, 2011, 11:00am MDT

As a non-profit organization, Mozilla has a strong commitment to personal privacy and empowerment. But after we released the last update to Firefox Beta for Android, many people started asking us why Firefox needed access to their phone numbers.

Firefox does not access users’ phone numbers, but it was clear that we needed to address this concern. Where did these questions come from? Here’s the first thing users saw when installing or updating Firefox Beta in the Android Market:

The “Phone Calls” permission was added in the last update to Firefox Beta (but has been since been removed, as I’ll explain below). When users installed that update and tapped on “Phone calls” for more information, they saw this:

Why did Firefox Beta ask for this permission? Firefox did not ever access phone numbers, serial numbers, or phone calls. But it did have code to detect the type of network connection: 2G, 3G, 4G, Wi-Fi, and so on. Firefox or add-ons could use this code to change settings automatically based on network type, for example to use less data on mobile networks.

Unfortunately, this required permission to READ_PHONE_STATE, which also grants access to very sensitive data. We knew this would worry some users, so we immediately started working on explaining how and why Firefox uses various permissions. We now have this information on our support site and will link to it from our Android Market page.

But the reaction to the new permission in Firefox Beta was so strong that we decided to remove that permission completely, along with the code that used it. Now when you go to the Android Market to install Firefox Beta, it will no longer ask to read “phone state and identity.”
Thoughts on Android Permissions
Permissions on Android and similar platforms are not perfect, but they do give users some useful tools to protect themselves. When an app requests only minimal permissions, users know it can do only limited damage if it is buggy or malicious. Recent versions of Android also have well-written explanations of each permission to help users make decisions.

But when an app requests lots of permissions, users have a tough choice. They can grant the permissions, or not use the app at all. This is especially bad for permissions like READ_PHONE_STATE that are needed for some reasonable features but also provide access to sensitive data. Eventually, most people probably get used to granting whatever permissions are requested, especially for apps like Facebook and Netflix that provide unique access to popular services.

Making permissions finer-grained might help (for example, separating “Read phone number” from “Read connection type”), but would also mean longer lists of permissions. That could make users even less likely to read and understand them. Explanations from developers can also help, but only if users trust them to tell the truth. Allowing users to grant or deny individual permissions (perhaps only at the time the app needs them) might help too, or it might just train users to always grant permissions so that apps will stop nagging them.

Aside from these overall design issues, there are also bugs in the developer documentation, and a bug that causes old permissions to stick around even after updating to a new version that doesn’t need them. These little bugs make it harder for developers to do the right thing. Some researchers at UC Berkeley have analyzed the Android source code to produce tools and documentation that fill in some of the gaps for developers.

The good news is that some users are paying attention, and those users make things better for everyone by pressuring developers (like us!) to remove invasive permissions. If you’re one of the Firefox fans who wrote to us about the new permissions in Firefox Beta, thank you! We appreciate it.

2 Star

Making Knuth’s wish come true: the x32 ABI

Posted: September 26th, 2011, 5:34pm MDT by Josh

Several years ago (though I can’t say exactly how many since it’s not dated) Knuth made the following complaint:

A Flame About 64-bit Pointers

It is absolutely idiotic to have 64-bit pointers when I compile a program that uses less than 4 gigabytes of RAM. When such pointer values appear inside a struct, they not only waste half the memory, they effectively throw away half of the cache.

The gcc manpage advertises an option “-mlong32″ that sounds like what I want. Namely, I think it would compile code for my x86-64 architecture, taking advantage of the extra registers etc., but it would also know that my program is going to live inside a 32-bit virtual address space.

Unfortunately, the -mlong32 option was introduced only for MIPS computers, years ago. Nobody has yet adopted such conventions for today’s most popular architecture. Probably that happens because programs compiled with this convention will need to be loaded with a special version of libc.

Please, somebody, make that possible.

I always thought this made a lot of sense. People have asked distro-makers for this before without a lot of success, but it looks like this is now being worked on by high-profile people in the Linux community. It is called The x32 ABI (see the LWN coverage for a more digestible description). It’s exciting because in some benchmarks this can outperform the x86-64 ABI by 10% or more. It’s a tradeoff — if you don’t need to address more than 4GB of memory, you can get faster programs because smaller pointers have better cache utilization. You’ll use less memory too.

This could have been done in a way that operated nearly the same as “compatibility mode” (ie. running 32-bit binaries on a 64-bit CPU/OS), which would have required only minimal changes to the kernel/toolchain. But it looks like their plans are more ambitious: they want to be able to use the optimized SYSCALL64 instruction (which is “much faster” than int 0x80 according to H. Peter Anvin), and they’re looking at fixing other problems like 32-bit time_t. So it’s a more substantial effort, but it looks like there’s significant interest and momentum behind this.

Thinking about how this would affect upb, my impression is that I could use my x86-64 JIT unmodified with x32, since it appears to have all of the same calling conventions. It has the same set of callee-save registers and the same set of registers for parameter transfer, and I think these are the main things upb’s JIT-ted code depends on.
Beating the compiler

Posted: September 17th, 2011, 4:08pm MDT by Josh
It’s been a while since I’ve posted about upb, but I’ve been busy improving it! I think the biggest achievement I can mention is that the core upb APIs (upb_handlers, upb_def, and upb_bytestream/upb_bytesink) are converging to the point where I’m comfortable with people starting to experiment with them. I’m not promising they won’t change at all, but I’m a lot more confident in their overall structure and semantics than I have been previously.

Notably, I think upb’s deserialization is ready for casual and experimental use. Definitely don’t trust any data to it until it’s better tested, though. I won’t be releasing until things have converged a bit more (and are better tested).

I gave a talk about upb at Google yesterday that was well-received. One question that comes up is “how are you beating the generated code from the protobuf compiler?” For the record, my JIT appears to be about 25% faster than Google’s protobuf release on a completely apples-to-apples test. It is a bit surprising, since my code has basically the same structure as protobuf’s generated C++ — I don’t invent any new optimizations or anything like that. I think it really comes down to generating better assembly than gcc is.

We live in a day and age where common wisdom is that you can’t beat a good C++ compiler, or at least not by much, and I think this is probably true for 99% of use cases. But I was first inspired to think that I could beat the C++ compiler in this case by reading this mailing list post from Mike Pall where he explains why you can still beat the compiler for interpreter main loops. A protobuf parser is surprisingly similar to a byte-code interpreter main loop, so I thought I’d give it a shot.

Below is just the simplest example I could dig up of a side-by-side comparison of my code vs. the compiler’s. What follows is the code to parse a single fixed64 value:
```
  ; upb JIT assembly:
  mov    rdx,QWORD PTR [rbx+0x2]    ; load fixed64 val out of buffer
  add    rbx,0xa                    ; advance buffer by 10 (2 for tag)
  mov    QWORD PTR [r12+0x40],rdx   ; store fixed64 value in message
  or     BYTE PTR [r12+0x1],0x4     ; set hasbit
  cmp    rbx,QWORD PTR [r15+0xaf8]  ; check for end-of-buffer
  jae    
  mov    rcx,QWORD PTR [rbx]        ; load next tag
  cmp    cx,0x1b0                   ; next field+wt in order?
  je     
```
There’s not a lot left to cut away here. Compare with protobuf/gcc-generated code:
```
  ; protobuf/gcc code:
  mov    ecx,DWORD PTR [rbx+0x10]   ; load buffer end
  mov    rax,QWORD PTR [rbx+0x8]    ; load buffer
  sub    ecx,eax                    ; if (buffer_end_ - buffer_ 
  mov    rax,QWORD PTR [rax]        ; load fixed64 val
  mov    rdx,QWORD PTR [rbp-0x48]   ; load this
  mov    QWORD PTR [rdx],rax        ; store fixed64 val in this
  add    QWORD PTR [rbx+0x8],0x8    ; advance buffer
  or     DWORD PTR [r12+0x74],0x800 ; set hasbit
  mov    rdx,QWORD PTR [rbx+0x10]   ; load buffer end
  mov    rax,QWORD PTR [rbx+0x8]    ; load buffer
  mov    ecx,edx
  sub    ecx,eax                    ; if (buffer_end_ - buffer_ 
  cmp    BYTE PTR [rax],0xb0        ; check first byte of next tag
  jne    
  cmp    BYTE PTR [rax+0x1],0x1     ; check second byte of next tag
  jne    
```
There is some poor register allocation going on here — gcc is repeatedly loading buffer_ and buffer_end_ even though it has plenty of registers to play with. All in all, gcc is generating over twice the number of instructions, over twice the number of loads, and more stores too. Note that this is taken from the middle of a very large C++ function with a big switch statement and a bunch of gotos, and I think it’s just difficult for a compiler to do good register allocation under these constraints.

If the C++ compiler could know the difference between fast paths and slow paths and do the register allocation solely for the fast paths (spilling everything for the slow paths) it might have a better shot. But still I think it’s just a hard problem.

Comments for Josh the Outspoken

Comment on Addicted to hardware: new toy on the way by Alexandre Abreu

Posted: September 5th, 2011, 1:00pm MDT by Alexandre Abreu

have you seen this a bit more expensive but a bit better gadget:

[trimslice.com]

w/ a NVIDIA Tegra2 as CPU …
Comment on Addicted to hardware: new toy on the way by Josh

Posted: August 26th, 2011, 6:47pm MDT by Josh

I like it! Unfortunately it doesn’t work too well as a headless always-on box for two reasons:

1. it loses its wireless connection sometimes and doesn’t automatically re-establish it for some reason. If I log in to it physically and click the wireless network again everything is fine, but that’s no good if you were planning to use it headless.

2. it doesn’t want to boot up unless it’s connected to a display. so you can’t boot it headless.

For these two reasons I don’t use it as much as I was expecting. But when I do use it it does just what I was hoping for.
Comment on Addicted to hardware: new toy on the way by Alexandre Abreu

Posted: August 24th, 2011, 1:37pm MDT by Alexandre Abreu

Any feedback on the EFIKA MX Smarttop?

Planet Matt

Bookmarks: Wood Tape

Posted: August 5th, 2011, 2:20pm MDT

This story convinced me to remind my own four-year-old about the wooden table she'd been wanting to build and paint, but that I hadn't got around to helping with.
Bookmarks: Wood Tape

Posted: August 2nd, 2011, 6:46pm MDT

This story convinced me to remind my own four-year-old about the wooden table she'd been wanting to build and paint, but that I hadn't got around to helping with.

Comments for Josh the Outspoken

Comment on Torn over the C++ question by Rodrigo Moraes

Posted: July 28th, 2011, 5:44am MDT by Rodrigo Moraes

If you’d be starting today, do you think it could benefit from using go? Also, could you please drop some general considerations about golang?

Planet Matt

Photos: Seattle in the sun and rain

Posted: July 27th, 2011, 5:06pm MDT
Photos: Ponies

Posted: July 27th, 2011, 5:06pm MDT

Captive Chincoteague ponies outside the Refuge Inn on Chincoteague island.
Bookmarks: How I Learned To Play Guitar

Posted: July 26th, 2011, 12:10pm MDT

"The lessons confused me, because though I enjoyed making progress, I didn’t see myself heading to that place of performing folk songs on the back porch, and I was completely unaware that there might be multiple valid approaches to an instrument. I just knew that what I was experiencing was not what I had envisioned."
Bookmarks: How I Learned To Play Guitar

Posted: July 26th, 2011, 12:10pm MDT

"The lessons confused me, because though I enjoyed making progress, I didn’t see myself heading to that place of performing folk songs on the back porch, and I was completely unaware that there might be multiple valid approaches to an instrument. I just knew that what I was experiencing was not what I had envisioned."

Comments for Josh the Outspoken

Comment on Books About Parsing by Rodrigo Moraes

Posted: July 20th, 2011, 7:50pm MDT by Rodrigo Moraes

jb: the PDF is from the first edition, from 1990. The second edition listed aboveis from 2007 and is not available as PDF.

Planet Matt

Bookmarks: Lisa Bloom: How to Talk to Little Girls

Posted: July 15th, 2011, 5:56pm MDT

"I always bite my tongue when I meet little girls, restraining myself from my first impulse, which is to tell them how darn cute/ pretty/ beautiful/ well-dressed/ well-manicured/ well-coiffed they are."
Bookmarks: Lisa Bloom: How to Talk to Little Girls

Posted: July 15th, 2011, 5:56pm MDT

"I always bite my tongue when I meet little girls, restraining myself from my first impulse, which is to tell them how darn cute/ pretty/ beautiful/ well-dressed/ well-manicured/ well-coiffed they are."
Journal: Meet the authors

Posted: June 27th, 2011, 9:56pm MDT

A few months ago we stayed at a small bed-and-breakfast while visiting family in Portland, Oregon. Two of our fellow guests were a couple with their own self-published comic book imprint. I had fun talking to them about their work, especially since I've been enjoying DMZ which is very similar to their latest title American Terrorist. (As a bonus, our kids are the same ages and managed to entertain each other nicely for part of our trip.) If you want to check out their work, you get the first issue for free as a PDF or from WOWIO or Comixology. The other issues are $1 each.

In other "self-published books by people I'm vaguely acquainted with" news, David D. Friedman (known to many of you as "patrissimo's dad") has published his second fantasy novel for the Kindle, and his medieval cookbook through Amazon's print-on-demand service. I've read and enjoyed both of his novels and several of his economics books; I haven't looked at the cookbook yet. (He also wrote a bit about the writing process and about self-publishing, if you want the behind-the-scenes view.)

Comments for Josh the Outspoken

Comment on Draft Syntax by Peter Lund

Posted: May 28th, 2011, 8:56am MDT by Peter Lund

” *(some-str)”

Wouldn’t “+(some-str)” be better for 1..n occurrences?

(apologies if this triple-posts … connection trouble)
Comment on upb status and preliminary performance numbers by John "Z-Bo" Zabroski

Posted: May 16th, 2011, 5:04pm MDT by John "Z-Bo" Zabroski

Interesting. David Ryan’s Argot project uses a JITer, and his explanation of using a JITer is not simply the reasons you give here, although they are bonus points for the Argot MCG (Micro Code Generator) which outputs C code.

The main idea behind Argot was resource consumption on embedded systems and to allow bootstrapping systems from nothing, like the kind the 6lowpan (IPv6 over Low power Wireless Personal Area Networks) working group were concerned with. David was a member of that committee, although somebody else’s ideas ultimately won out.

Something for you to blog about in the future, I guess!
Comment on upb status and preliminary performance numbers by Josh

Posted: May 9th, 2011, 12:01pm MDT by Josh
Will, that’s an interesting question but I doubt that Hotspot could generate code that is as good as mine. Not because I think I’m better at this than the Hotspot authors, but because I am compiling directly from a description of the .proto file into machine code. In the Hotspot case, the flow is more like:
```
  .proto file -> Java -> JVM byte-code -> machine code
```
The fact that you have to go through Java and JVM byte-code first means that you are limited by what can be expressed in Java. For example, Java doesn’t even have unsigned types!

In fact I think the JIT part is a bit of a distraction, because upb isn’t actually using any run-time information that isn’t available as part of the .proto file to begin with. So even the C++ protobuf compiler (which works ahead-of-time) has just as much information as upb’s JIT does. I’m JITting for convenience, not because I’m using any runtime information (well, at least not yet .

2 Star

Using dtrace on OS X to debug a performance problem

Posted: May 8th, 2011, 2:28pm MDT by Josh
I recently ported upb’s table-based decoder to use setjmp/longjmp-based error handling. I did this largely for code simplicity and readability, so that the non-error code-paths didn’t have to check for errors all the time. But unfortunately I noticed a dramatic 75% performance decrease. What was going on?

A profile in Shark showed the majority of my time being spent in system calls, but didn’t make it clear which system calls were involved. It sounded like a job for dtrace.

I used the following script to dump all the system calls that were being issued by my process:
```
syscall:::entry
/pid == $target/
{
     @[probefunc] = count();
} 
```
I then ran this script on both my old (fast) benchmark and the new (unexpectedly slow) one:
```
$ sudo dtrace -c ./old-benchmark -s trace.d

  exit                                             1
  fstat64                                          1
  ioctl                                            1
  write_nocancel                                   1
  munmap                                          93
  mmap                                            94
  getrusage                                     7728    

$ sudo dtrace -c ./new-benchmark -s trace.d

  exit                                             1
  fstat64                                          1
  ioctl                                            1
  munmap                                           1
  write_nocancel                                   1
  getrusage                                      438
  sigaltstack                                 111773
  sigreturn                                   111774
  sigprocmask                                 223546
```
Sure enough, the new version is making a ton of system calls that weren’t happening before, and they appear to be signal related. I investigated the manpage of setjmp/longjmp and found:

The setjmp()/longjmp() pairs save and restore the signal mask while _setjmp()/_longjmp() pairs save and restore only the register set and the stack. (See sigprocmask(2).)

The sigsetjmp()/siglongjmp() function pairs save and restore the signal mask if the argument savemask is non-zero; otherwise, only the register set and the stack are saved.

I replaced my calls to setjmp/longjmp with sigsetjmp/siglongjmp, passing 0 for my signal mask, and the performance problems went away. Score one for dtrace.

Comments for Josh the Outspoken

Comment on upb status and preliminary performance numbers by Will

Posted: May 7th, 2011, 11:30am MDT by Will

If JITting helps so much, how does the java implementation of protobufs compares when run under Hotspot? I’m guessing much slower, but I wonder if that has to be the case.
Comment on EINTR and PC loser-ing (The “Worse Is Better” case study) by Brian Maher

Posted: April 30th, 2011, 8:12am MDT by Brian Maher

Hi Josh,

I found this blog entry really interesting… I hadn’t thought about the fact that the PC losering problem is really about signals.

One example of where EINTR is important to me is when doing graceful server shutdowns. In particular, if I have a SIGTERM handler that sets a global “shutdown” variable, then I need to check if accept() returned EINTR and if so, did the “shutdown” global variable get set.

I like your idea of having separate system calls that retry if EINTR is returned, but it is simple enough to write those functions in a user program. In fact, it is probably a good practice to wrap most system calls to deal with other notoriously difficult to deal with errors such as ENOMEM.
Comment on upb status and preliminary performance numbers by Josh

Posted: April 29th, 2011, 4:25pm MDT by Josh

Thanks for the correction Russ. I was misled by the Wikipedia page about JSON.

I think it is safe to say, though, that interpreting all JSON numbers as double-precision is sufficient to be a usable JSON parser, since that what common implementations do.
Comment on upb status and preliminary performance numbers by Russ Cox

Posted: April 27th, 2011, 10:38pm MDT by Russ Cox

You say that JSON only supports double-precision numbers but that’s not true. JSON only defines the syntax of its numbers, not the semantics. There is no limit to the size or precision of numbers formatted as JSON.
Comment on upb status and preliminary performance numbers by Buffalo

Posted: April 26th, 2011, 1:25pm MDT by Buffalo

Congrats Josh!
Comment on upb status and preliminary performance numbers by Phil

Posted: April 26th, 2011, 1:02pm MDT by Phil

That must be a pretty awesome feeling.

2 Star

upb status and preliminary performance numbers

Posted: April 26th, 2011, 12:18am MDT by Josh
The last few weeks have been very exciting for upb. On April 1 I checked in a JIT compiler for parsing protobufs, which one might think was an April Fool’s joke, but it’s real and the performance numbers so far have exceeded my expectations.
Why JIT?
Before I get to the numbers, I should explain what it even means for upb to have a JIT. If you’re not interested in the technical details, feel free to skip this section to go straight to the impacts and what this means for upb.

So why a JIT? After all, protobufs are not an imperative language. You can’t write an algorithm in a .proto file. You can’t apply compiler techniques like SSA, strength reduction, tracing, or really any of the things you’d expect from a JIT on a platform like the JVM. There is no byte code or intermediate representation to speak of. So why would upb have a JIT and what on earth would it do?

To review, a .proto file defines a schema for some messages. In this sense, it is comparable to JSON Schema or XML Schema.
```
message Person {
  required int32 id = 1;
  required string name = 2;
  optional string email = 3;
}
```
To support forwards and backwards compatibility, we don’t make any assumptions about which fields we’re going to see or in what order. Instead, on the wire each field is preceded by its tag number. Logically, the serialized version is something like:
```
- field number: 1, wire type: varint, value: 1
- field number 2, wire type: delimited, value: "Josh Haberman"
- field number 3, wire type: delimited, value: "jhaberman@gmail.com"
```
If you were going to write a parser for this, you might create a table of fields, keyed by field number, and a main parser loop that looks something like this:
```
while (!done) {
  int type = fields[get_field_number()];
  switch (type) {
    case WIRE_TYPE_VARINT:
      parse_varint();
      break;
    case WIRE_TYPE_DELIMITED:
      parse_delimited();
      break;
    // etc.
  }
}
```
We call this a “table-based parser.” We look up the type in a table of fields, and use that to branch to the correct value parsing function. There are minor variations on this, like using a function pointer instead of a switch statement, but it’s the same general idea. To see an actual program that implements this style of parser, check out my bare-bones 100-line protobuf parser that I wrote a few years ago: pb.c.

In a way this resembles a byte-code interpreter, and the serialized protobuf resembles byte-code. For example, see the main loop of the Lua interpreter. It has a similar pattern where it extracts an opcode and uses that to branch into a giant switch statement.

It turns out that if we generate code that is specific to one message type, and takes advantage of the fact that fields are usually encoded in order, we can beat this table-based parser significantly. Google’s main “protobuf” software has done this for a long time: their strategy is to generate C++ classes that are specific to a proto message type (like Person in the above example). This code has been highly tuned and optimized, and the generated parsing code looks something like this:
```
while((tag = input->ReadTag()) != 0) {
  field_number = GetFieldNumber(tag);
  wire_type = GetWireType(tag);
  switch(tag) {
    // optional float field1 = 1;
    case 1:
      if (wire_type != FLOAT_WIRE_TYPE) goto error;
      input->ParseFloat();
      // Saves the switch() and type check branch if fields are in order.
      if (input->ExpectTag(2, DOUBLE_WIRE_TYPE)) goto parse_field2;
      break;

    // optional double field2 = 2;
    case 2:
      if (wire_type != DOUBLE_WIRE_TYPE) goto error;
parse_field2:
      input->ParseDouble();
      // saves the switch() and type check branch if fields are in order.
      if (input->ExpectTag(3, INT32_WIRE_TYPE)) goto parse_field3
      break;

    // optional int32 field3 = 3;
    case 3:
    // [...]
  }
}
```
This code also has a big switch() statement at the top level, but the targets are field numbers instead of wire types. More importantly, each field’s case has a line like this:
```
if (input->ExpectTag(3, INT32_WIRE_TYPE)) goto parse_field3
```
This is key, because if the fields occur in order (which they usually do) then this branch will always be taken and it will be predicted correctly by the CPU. This can make a huge difference.

While this kind of C++ code generation has benefited Google’s protobuf software in speed, I always found it inconvenient from a dynamic languages perspective. What dynamic language user wants to have to generate C++ code and link that into their interpreter? We’re not talking about a single C++ extension that you compile once: every single message you define in a .proto file generates different C++ code. So you can’t just apt-get install libprotobuf-python and be done with it, you have to generate C++ for every message that your specific app wants to use.

This is where the JIT comes in. If we can generate machine code at runtime, we can get the speed benefits of the generated code without having to generate, compile, and link C++ into our interpreters or VMs. You could compile the library once, and after that you can dynamically load message definitions but still get the fastest possible parsing.
Preliminary results
I knew all of the theory I just described before I started writing a JIT. But theory is just that — theory. What would actually happen when I implemented a JIT?

The results exceeded my expectations. I still am being somewhat cautious, because there are so many dimensions to any performance equation, and because benchmarks are not guaranteed to correspond to real-world performance. That said, here are my preliminary results. You can reproduce these by obtaining upb from GitHub and running make benchmarks followed by make benchmark (you have to define -DUPB_USE_JIT_X64 to get the JIT).
```
Parsing an 80k protobuf into a data structure repeatedly,
calling Clear() between each parse.  (proto2 == Google protobuf)

proto2 table-based parser      38 MB/s
proto2 generated code parser   265 MB/s
upb table-based parser         340 MB/s
upb JIT parser                 741 MB/s
```
The results above are designed to be as apples-to-apples as possible. In other words, I disabled upb’s optimization that avoids memcpy() because Google’s protobuf doesn’t support it in its open-source release. I think the reason that even my table-based parser beats Google’s generated code here is because proto2 implements Clear() in a sub-optimal way that requires an extra pass over the message tree; see my comment in upb_msg.h for more info.

Things get even better if we allow ourselves to drop the constraint of being apples-to-apples with proto2. upb is capable of stream-based parsing like SAX, whereas proto2 only supports a DOM-based approach where you parse into a data structure. If we include the performance numbers for stream parsing, and for DOM-based parsing that avoids memcpy, we get:
```
upb table-based stream parser    420 MB/s
upb JIT no-memcpy() DOM parser   870 MB/s
upb JIT stream parser           1430 MB/s
```
If you’re like me when I first saw these numbers, your jaw is on the floor at seeing almost 1.5GB/s doing stream parsing of protocol buffers. At this point we are 5x proto2′s highly optimized generated code.

upb’s JIT isn’t complete and can’t handle all cases, but these performance numbers should still be valid because these benchmarks only used the parts of the format that are currently supported.
So when can I USE it? And can I help?
One reason I’ve avoided posting these extremely positive results so far is that I hate to get people excited about something that’s still not ready for users yet.

Adding support for the JIT required making extremely large and intrusive changes to the core interfaces, like this 3000 line refactoring that had to be completed before I could write even the first line of the JIT. I need to have the flexibility to change core interfaces like this still, because the design is still converging. So as much as I wish I could say it’s ready to go, I still need to hold off on a real release.

The good news is that the design is still making huge strides. In the last few weeks I’ve been refining my scheme for how upb will integrate into different VM’s and language runtimes, and I feel more confident than ever that the language bindings for Lua, Python, etc. will be some of the fastest extensions ever offered for this kind of functionality.

As a preview for where this is going, I think that upb will even be usable as a JSON library, offering speed that is as good or greater than any existing JSON libraries. JSON can be directly mapped onto a subset of protobufs (JSON only uses double-precision numbers) and the protobuf text format is already extremely similar to JSON. So all the work I’ve done to optimize memory layout and dynamic language object access should apply.

And while I’m really happy to get offers to help out, it’s still at a stage of design where I need to be doing most of the work. Working on upb generally involves pacing around my apartment deep in thought about all the requirements and use cases I want to satisfy and brainstorming a million different approaches until I converge on the the one that is the smallest, fastest, and most flexible. It’s hard work but I love it, and the more time that passes the more convinced I am that this is going to be big.

Comments for Josh the Outspoken

Comment on Gazelle/upb status by Ricky

Posted: April 25th, 2011, 3:17pm MDT by Ricky

Is there a reasonably stable version of upb that can be used in production? If it works well enough, I guess I could fix small issues myself if it comes to that… but how “done” is your implementation really? It sounds brilliant by the way.
Comment on EINTR and PC loser-ing (The “Worse Is Better” case study) by John "Z-Bo" Zabroski

Posted: April 25th, 2011, 11:14am MDT by John "Z-Bo" Zabroski

“Worse is better” systems incrementally improve by responding to user needs.

Are your conclusions meant to be generalized, and, if so, in what way?

For example, web browsers adopted the Robustness Principle and now we have the messy result of software that has to know how to interpret its formats, rather than distributing the format with the interpreter (Joel Spolsky’s rant titled “Martian Headsets” gives a good summary of why this is such a bad approach, and what we are stuck with). The web browser example is actually not mine; it is Alan Kay’s classic example of Doghouse and Pyramid architecture from his classic OOPSLA talk, “The Computer Revolution Hasn’t Happened Yet” (google it). He essentially argues, implicitly, the Smalltalk way is better — and before Tim Berners-Lee invented the Web Browser, Xerox PARC had an Object Browser built into the operating system via the Smalltalk system.
Comment on EINTR and PC loser-ing (The “Worse Is Better” case study) by Josh

Posted: April 24th, 2011, 12:37am MDT by Josh
Thanks for the comment! I agree that providing EINTR for applications that want it can possibly be useful. But I think the way it’s exposed in the API breaks the rule of “make simple things easy and hard things possible.” Since it’s very uncommon that a program wants to handle EINTR, I think it would be better to have separate versions of the functions (or a boolean parameter to these functions) that lets you enable EINTR only for calls where you want to handle it.
```
// Standard write() call.  Will never return EINTR.
ssize_t write(int fildes, const void *buf, size_t nbyte);
 
// Interruptible write() call, may return EINTR.
ssize_t interruptible_write(int filedes, const void *buf,
                            size_t nbyte);
```
As it’s written, since SA_RESTART is a property of the signal handler and not of the interruptible system call itself, all such system calls are potentially interruptible, so they all have to handle EINTR.

2 Star

ARM architecture is mired in oppressive legalese

Posted: April 18th, 2011, 11:36pm MDT by Josh

My Efika MX Smarttop came and I’ve had some fun compiling upb for it and trying it out. I don’t have a pressing ARM use case yet, but I want to work ahead a bit and get familiar with the architecture so that when I do want to program for it I’m not starting from scratch.

I went to the ARM website to get my hands on some documentation. I was prepared to buy physical books if that is their authoritative documentation, but it appears that they have PDFs for the architecture’s reference manuals.

So far so good — this is on par with Intel which has a nice and easily accessible website where you can download any of their manuals in like 5 seconds. I’ve downloaded them and referenced them a countless number of times.

But I get no such love from ARM. Strike 1: the ARM website won’t let you download the manuals unless you have registered first. Not cool ARM, not cool. I grudgingly give them my name, email address, company name, country, and state. And the website warns:

Note: We recommend using your business email address to ensure you can access all your relevant services

This is a vague but ominous warning that if you use a personal email address the registration might not work correctly.

But fine, I go through with the registration. Now I can download the manual, right? Turns out no: first I have to accept a EULA! It begins:

USER AGREEMENT FOR THE ARM ARCHITECTURE REFERENCE MANUAL

THIS AGREEMENT (” AGREEMENT “) IS A LEGAL AGREEMENT BETWEEN YOU (EITHER A SINGLE INDIVIDUAL, OR SINGLE LEGAL ENTITY) AND ARM LIMITED (“ARM”) FOR THE USE OF THE ARM ARCHITECTURE REFERENCE MANUAL. ARM IS ONLY WILLING TO PROVIDE ACCESS TO THE ARM ARCHITECTURE REFERENCE MANUAL TO YOU ON CONDITION THAT YOU ACCEPT ALL OF THE TERMS IN THIS AGREEMENT. BY CLICKING “I AGREE” OR BY DOWNLOADING OR OTHERWISE COPYING THE DELIVERABLES YOU INDICATE THAT YOU AGREE TO BE BOUND BY ALL THE TERMS OF THIS LICENCE.

This is all just to read some documentation. Not impressed. ARM, We’re off to a rocky start.
EINTR and PC loser-ing (The “Worse Is Better” case study)

Posted: April 18th, 2011, 2:57am MDT by Josh
Richard Gabriel’s 1989 essay Worse Is Better is a famous comparison between LISP and Unix/C that pops up from time to time and is guaranteed to spark a spirited discussion. The philosophical argument itself is not something I want to get into right now; I am interested in the technical content of the essay. What always bothered me about this paper is that I never fully understood Gabriel’s primary example of a dirty hack vs. “the right thing.”

His example is “the PC loser-ing problem,” which he describes thus:

Two famous people, one from MIT and another from Berkeley (but working on Unix) once met to discuss operating system issues. The person from MIT was knowledgeable about ITS (the MIT AI Lab operating system) and had been reading the Unix sources. He was interested in how Unix solved the PC loser-ing problem. The PC loser-ing problem occurs when a user program invokes a system routine to perform a lengthy operation that might have significant state, such as IO buffers. If an interrupt occurs during the operation, the state of the user program must be saved. Because the invocation of the system routine is usually a single instruction, the PC of the user program does not adequately capture the state of the process. The system routine must either back out or press forward. The right thing is to back out and restore the user program PC to the instruction that invoked the system routine so that resumption of the user program after the interrupt, for example, re-enters the system routine. It is called “PC loser-ing” because the PC is being coerced into “loser mode,” where “loser” is the affectionate name for “user” at MIT.

The MIT guy did not see any code that handled this case and asked the New Jersey guy how the problem was handled. The New Jersey guy said that the Unix folks were aware of the problem, but the solution was for the system routine to always finish, but sometimes an error code would be returned that signaled that the system routine had failed to complete its action. A correct user program, then, had to check the error code to determine whether to simply try the system routine again. The MIT guy did not like this solution because it was not the right thing.

The New Jersey guy said that the Unix solution was right because the design philosophy of Unix was simplicity and that the right thing was too complex.

When I read this I always had a burning desire to know: how did the story end? How do modern operating systems resolve this problem — the “dirty hack” way or the “right way?” What part of our modern POSIX interfaces are affected by this question?

There are several things that never made sense to me about this example. First of all, why would you need to abort a system call just because an interrupt occurred? I investigated the Linux source and it seems quite clear that interrupt handlers can return to either the kernel or userspace — whichever was running when the interrupt fired. So I don’t see why you’d need to “coerce” the system into “loser mode” at all.

But let’s suppose you accept this as a given — we will assume that when a hardware interrupt occurs, you must exit to user mode. I still don’t see the difficulty in automatically re-invoking the system call. It’s true that invoking the system routine is a single instruction, but why is it that “the PC of the user program does not adequately capture the state of the process,” as Gabriel’s essay states? What other process state do we need to capture? The registers must already be saved when the syscall is entered, because they must be restored even with a completely normal syscall return. So if we want to re-invoke the system routine, it should be as easy as simply re-executing the instruction that made the system call. Right?

The whole example confused me quite a lot until I had the idea to replace “interrupt” in the above description with “signal.” This is not such a stretch, since signals are essentially user-space software interrupts. With this small change, everything started to make a lot more sense. If a signal was delivered to a process that was currently inside a system call, that signal handler could invoke a system call itself, which would cause us to re-enter the kernel. I could easily see how the complexity of dealing with this could have led early UNIX implementors to simply abort the original system call before delivering the signal.

But this is only speculation about what UNIX was like in the mid to late 80s when “Worse is Better” was written. I could be completely off the mark in this analysis — maybe returning to the kernel from a hardware interrupt handler really wasn’t implemented at that time. Or maybe saving user state really was difficult for some reason. I’d love to hear from anyone who has more historical context about this. But the essay contains an important clue that seems to reinforce my speculation that it’s actually about signals.
UNIX and EINTR
If we look closely at the “Worse is Better” essay, we get a strong clue about what the Unix guy in the story might have been talking about:

The New Jersey guy said that the Unix folks were aware of the problem, but the solution was for the system routine to always finish, but sometimes an error code would be returned that signaled that the system routine had failed to complete its action. A correct user program, then, had to check the error code to determine whether to simply try the system routine again.

As someone who has done a lot of Unix system-level programming, this sounds to me like it must be describing EINTR, the error code in Unix that means “Interrupted system call.” To give a quick description of EINTR I’ll enlist the help of my trusty copy of “Advanced Programming in the Unix Environment” by W. Richard Stevens:
A characteristic of earlier UNIX systems is that if a process caught a signal while the process was blocked in a “slow” system call, the system call was interrupted. The system call returned an error and errno was set to EINTR. This was done under the assumption that since a signal occurred and the process caught it, there is a good chance that something has happened that should wake up the blocked system call.

[...]

The problem with interrupted system calls is that we now have to handle the error return explicitly. The typical code sequence (assuming a read operation and assuming that we want to restart the read even if it’s interrupted) would be
```
again:
  if ((n = read(fd, buf, BUFFSIZE)) 
```
This sounds an awful lot like the the New Jersey guy's approach from the story, which required a correct program "to check the error code to determine whether to simply try the system routine again." And there's nothing else in Unix that I've ever heard of that's anything like this. This must be what the New Jersey guy from the story was talking about!

But note that in W. Richard Stevens' explanation this isn't some dirty hack! It's not a case of cutting corners that is justified by favoring implementation simplicity over interface simplicity. Stevens describes it as a deliberate design decision that gives users the capability to abort a long-running system call if you catch a signal in the meantime. Now you could easily see this as a rationalization of a dirty hack ("it's not a bug, it's a feature!"), but it certainly seems plausible that if you catch a signal while you're blocked on a long system call, the signal might make you decide that you don't want to wait for the long system call any more. Indeed, Ulrich Drepper claimed in 2000 that "Returning EINTR is necessary for many applications," though it would have been helpful if he had expanded on this point by giving some examples.

Of course, the price we have paid for this capability is that we have to wrap all of our potentially long system calls in a loop like the example above. If we don't, our system calls can start failing and causing program errors whenever we catch a signal. You may think that you don't use any signals yourself, but are you sure that none of your libraries do? On the flip side, if you're implementing a library you can never know if the main application will use signals or not, so any library that wants to be robust will have to wrap these system calls in a retry loop.

Since the vast majority of programs will always want their system calls to continue even when a signal is received, 4.2BSD (released in 1983) implemented support for automatically retrying most system calls that could previously fail with EINTR. To me this sounds exactly like what the MIT guy in Richard Gabriel's story was saying is "the right thing." In other words, Berkeley UNIX was already doing "the right thing" five years before "Worse is Better" was written!

Modern POSIX APIs allow both behaviors (either restarting the system call automatically or returning EINTR) -- this is controlled by the SA_RESTART flag. The following program illustrates both behaviors:
```
#include 
#include 
#include 
#include 
#include 

void doread() {
  char buf[128];
  printf("doing read() into buf %p\n", buf);
  ssize_t ret = read(STDIN_FILENO, buf, sizeof(buf));
  if (ret  1) ? SA_RESTART : 0;
  sigaction(SIGHUP, &action, NULL);

  doread();
  return 0;
}
```
Here are the results of running the program three different times. I've bolded the parts where I typed to give the program input on stdin. You can also see where I sent the program a SIGHUP.
```
$ ./test
doing read() into buf 0x7ec7959c
INPUT FROM TERMINAL
read() for buf 0x7ec7959c returned data: INPUT FROM TERMINAL
$ ./test
doing read() into buf 0x7ef6659c
received signal 1
doing read() into buf 0x7ef66204
INPUT FROM TERMINAL
read() for buf 0x7ef66204 returned data: INPUT FROM TERMINAL
read() for buf 0x7ef6659c returned error: Interrupted system call
$ ./test give_me_sa_restart
doing read() into buf 0x7eb7657c
received signal 1
doing read() into buf 0x7eb761e4
INPUT FROM TERMINAL
read() for buf 0x7eb761e4 returned data: INPUT FROM TERMINAL
INPUT FROM TERMINAL AGAIN
read() for buf 0x7eb7657c returned data: INPUT FROM TERMINAL AGAIN
```
Conclusion
You might ask "why all the fuss over a little example?" As I mentioned, my primary motivation in researching all of this was to get to the bottom of this issue and understand how it plays out in modern operating systems.

But if we were going to take all of this information and reflect on the "Worse is Better" argument, my personal observations/conclusions would be:
- The "worse" system (Unix) did indeed do "the right thing" eventually, even if it didn't at first. "Worse is better" systems incrementally improve by responding to user needs. Since users got tired of checking for EINTR, the "worse" system added the functionality for addressing this pain point.
- The whole thing did leave a rather large wart, though -- all Unix programs have to wrap these system calls in an EINTR retry loop unless they can be absolutely sure the process will never catch signals that don't have SA_RESTART set. So there is a price to pay for this incremental evolution.

Planet Matt

Bookmarks: http://www.ipuz.org/

Posted: April 14th, 2011, 3:25pm MDT

An extensible JSON standard for representing crosswords and other puzzles.
Bookmarks: http://www.ipuz.org/

Posted: April 14th, 2011, 3:25pm MDT

An extensible JSON standard for representing crosswords and other puzzles.
Bookmarks: Android Developers Blog: Android Browser User-Agent Issues

Posted: April 12th, 2011, 4:06pm MDT

"This posting describes some issues when browsing websites with mobile variants using large-form-factor Android devices."
Bookmarks: Android Developers Blog: Android Browser User-Agent Issues

Posted: April 12th, 2011, 4:06pm MDT

"This posting describes some issues when browsing websites with mobile variants using large-form-factor Android devices."
Bookmarks: netpoetic.com > unicode

Posted: April 11th, 2011, 6:09pm MDT

"i made video called 'unicode'. it shows all displayable characters in the unicode range 0 – 65536 (49571 characters). one character per frame."
Bookmarks: netpoetic.com > unicode

Posted: April 11th, 2011, 6:09pm MDT

"i made video called 'unicode'. it shows all displayable characters in the unicode range 0 – 65536 (49571 characters). one character per frame."

2 Star

Addicted to hardware: new toy on the way

Posted: March 31st, 2011, 2:50am MDT by Josh

I’m addicted to hardware. I can’t stop thinking about all of the CPUs that currently exist, how they compare to each other, and how to write the fastest possible code on them. (Actually I want to learn FPGA programming too, in case they ever start bundling FPGAs with computers).

A year and a half ago I bought a SheevaPlug which is a little ARM computer that’s hardly bigger than a wall wart AC adapter. It has USB and Ethernet and that’s about it (no display). Unfortunately when I tried to start playing with it again tonight I discovered it was bricked thanks to a faulty power supply, which is apparently a very common problem with SheevaPlugs.

I was a bit sad about this, and I started looking for alternatives. I wanted something that runs on a non-x86 architecture and that I could stick in a closet and SSH to. I discovered that there is a new form factor of computers known as a nettop. I found a nettop that appears to fulfill my wishes perfectly: the EFIKA MX Smarttop made by a company called Genesi. It’s surprisingly capable for $129: it’s powered by an ARM Cortex-A8 800MHz CPU, has 512MB RAM, 8GB of flash, Ethernet, WiFi, Bluetooth, USB, and 720p video output through HDMI (with hardware-accelerated video decoding). Not bad for $129! And it has a maximum power consumption of 15W.

So I ordered one — will look forward to seeing if it is really all it’s stacked up to be!
When a compiler’s slow code actually bites you

Posted: March 19th, 2011, 4:26pm MDT by Josh
A few days ago I posted GCC: the impressive and the disappointing where I looked at some cases where GCC produces not-quite-optimal code. One of the comments on that post was (emphasis mine):

So, it seems like there is a much better way to give the compiler a shot at doing the right thing: [snip suggestion]. I think you will find the compiler will generate quite efficient code in this case, particularly if you look at the real execution overhead, rather than what the assembler looks like.

This is a common attitude I encounter when I am discussing my attempts to optimize my protocol buffer decoding library upb. Programmers love to tell other programmers that they are prematurely optimizing, and most of the time they’re right. I’m sure to some people it seems ludicrous that I would be looking at assembly language output to determine whether it is efficient enough. For 99.99% of programs, it would be. But I’m working in one of those rare domains where it actually matters. And today I encountered pretty convincing evidence that the compiler’s bad code is actually affecting me.

The compiler’s bad code in this case is an example of a bug I previously filed on GCC: struct returned by value generates useless stores. Though I had previously observed that bug only by inspecting assembly language output, today I had it show up on an actual profile as clear as day. Here is a screenshot from Shark (click to get full-size):

To summarize, the compiler took the code:
```
typedef struct {
  upb_flow_t flow;  // An enum defined elsewhere.
  void *closure;
} upb_sflow_t;

upb_flow_t upb_dispatch_startsubmsg([...]) {
  // [...]
  upb_sflow_t sflow = f->cb.startsubmsg([...]);
  if (sflow.flow != UPB_CONTINUE) {
    // [...]
  }
```
…and turned that function call/test into this awful machine code (here in its Intel-syntax form):
```
  call   QWORD PTR [r12 + 16]
  mov    DWORD PTR [rbp - 64], eax
  mov    QWORD PTR [rbp - 56], rdx
  mov    rax, QWORD PTR [rbp - 64]    ; loads rax with data it already has.
  mov    QWORD PTR [rbp - 48], rax    ; stores rax into the stack a second time.
  mov    QWORD PTR [rbp - 40], rdx    ; stores rdx into the stack a second time.
  mov    edx, DWORD PTR [rbp - 48]    ; loads edx with data already in rax.
  testl  edx, edx
```
..and then (this is the important part) in an actual profile it shows up as being 43.4% of the execution time of a hot function in my program.

This is not a slam against the GCC developers. GCC is a big and complex piece of software, and they have to prioritize all sorts of different bugs, feature requests, new hardware, etc.

This is just a reminder to those who jump to dare-I-say “premature” conclusions about what is premature optimization: some of us really are working in domains where things like virtual function overhead, branch predictability, and the efficiency of the compiler’s code make a difference.
GCC: the impressive and the disappointing

Posted: March 4th, 2011, 12:58am MST by Josh
In my work on upb I’ve looked at a lot of compiler-generated assembly code. I frequently want to know how GCC will compile a certain block of code, so I’ll write a little test program in C and use objdump to look at the object file.

Over two years of doing this, I’ve had many moments where I am pleasantly surprised at how smart the compiler is being. C compilers can figure out a lot. I demonstrated a few examples of this on stackoverflow.com — see here and here.

But once in a while I am disappointed, because GCC doesn’t figure something out that I really wish it could. For example, I have a situation where I have a callback interface, but one of the parameters to my callback is something that a lot of clients don’t need. For convenience, I want to let them register a callback that doesn’t take the final parameter. But to be ANSI C, I can’t cast between the two callback types, I need to store the two possible callback types in a union. My quick test looked like this:
```
// My two callback types -- the second takes an extra parameter.
typedef union {
  void (*f1)(int);
  void (*f2)(int, int);
} funcs;

// I store "which" separately to track which type of callback was registered.
void foo(funcs f, int which) {
  int a = 5, b = 10;
  if (which) {
    f.f1(a);
  } else {
    f.f2(a, b);
  }
}
```
Now on x86-64, parameters are passed in registers (thank god!), so there’s no actual reason to branch here. You might as well just always put both values in registers, because if we were calling “f1″ it will just ignore the register that’s holding the second parameter, or overwrite it which is fine too. The only reason we put the “if” in the C code was to be ANSI C compilant — according to the standard you can’t cast between function pointer types.

But alas, GCC didn’t figure this out:
```
0000000000000000 :
   0:	85 f6                	test   esi,esi
   2:	48 89 f8             	mov    rax,rdi
   5:	75 11                	jne    18 
   7:	be 0a 00 00 00       	mov    esi,0xa
   c:	bf 05 00 00 00       	mov    edi,0x5
  11:	ff e0                	jmp    rax
  13:	0f 1f 44 00 00       	nop    DWORD PTR [rax+rax*1+0x0]
  18:	bf 05 00 00 00       	mov    edi,0x5
  1d:	ff e0                	jmp    rax
```
Notice it put a branch in there (“jne”) just to avoid putting the value 10 (0xa) in register esi for the one-argument path.

It’s even worse if I give both functions one parameter, but of different types:
```
typedef union {
  void (*f1)(int);
  void (*f2)(long);
} funcs;

void foo(funcs f, int which) {
  int a = 5;
  if (which) {
    f.f1(a);
  } else {
    f.f2(a);
  }
}
```
In this case, GCC again generates a branch, but both paths have identical code in them!
```
0000000000000000 :
   0:	85 f6                	test   esi,esi
   2:	48 89 f8             	mov    rax,rdi
   5:	75 09                	jne    10 
   7:	bf 05 00 00 00       	mov    edi,0x5
   c:	ff e0                	jmp    rax
   e:	66 90                	xchg   ax,ax
  10:	bf 05 00 00 00       	mov    edi,0x5
  15:	ff e0                	jmp    rax
```
Both branches simply move 5 into edi and jump to rax. There is absolutely no reason to branch here. Sigh.

C compilers are smart, but have their limits. Another thing this demonstrates is how programming in C can be a bit constraining compared to assembly language. The only reason I’m jumping through these hoops to begin with is that C has very strict rules about pointer conversion: you can’t just go around casting one function pointer type to another, because you’ll get undefined behavior. But if you’re programming in assembly there is no undefined behavior, no worrying about aliasing, etc. The code I’m trying to get C to generate in a standards-compliant way would be trivial to write in assembly language directly.

Of course in that case I’d have to implement the assembly language on every platform I wanted to target. In the end I’ll probably use the branchy version that the compiler will generate; the branch will probably predict pretty well, and more importantly for the real fast path for protobuf decoding I have a JIT on the way…

Planet Matt

Bookmarks: Halle Hula | MeFi Music

Posted: February 18th, 2011, 4:47pm MST

"When Elvis Met Leonard."
("What the hell is wrong with you?")
Bookmarks: Halle Hula | MeFi Music

Posted: February 18th, 2011, 4:47pm MST

"When Elvis Met Leonard." ("What the hell is wrong with you?")
Buzz: Buzz by Matt Brubeck from Buzz

Posted: February 15th, 2011, 2:48pm MST

A while ago I disconnected my Twitter stream from Buzz, since it was kind of chatty and of limited interest and I figured you could just follow me on Twitter if you wanted to read it. Today I am reconnecting it, but now you can click the arrow on any tweet and choose "Mute Twitter posts from Matt Brubeck" if you don't want to see them here. (Thanks, Google Buzz team.)

2 Star

Gazelle being integrated into a syntax aware editor

Posted: February 14th, 2011, 12:53am MST by Josh

I’ve been meaning to post this for several days now: I was very excited to hear on the gazelle-users mailing list that Gazelle is being integrated into a syntax-aware editor called Kod. See the screencast for a preliminary demo.

This is very exciting for me, because the premise that a text editor could use a real parser as you type was one of my major motivations for creating Gazelle. Most text editors clever pattern matching for the syntax highlighting, but since they aren’t real parsers they can get confused. More importantly, they can’t give you true syntax-aware completion. The notable exception is Java editors like Eclipse, which do this quite well, and I hear that Microsoft’s IDEs are good at it too. But I always wanted an editor where you could add new languages easily and have more ability to tinker with how this syntax information is used. I think it could open up so many doors.

Given this, I was very excited when Rasmus Andersson from the Kod project wrote to the gazelle-users mailing list asking if Gazelle was a dead project, or if it was something he and his team should invest effort into integrating with Kod. I replied that it was definitely not a dead project, but that it is still subject to major change. That didn’t seem to dissuade him though, and I look forward to learning from their needs and folding their improvements back into Gazelle.

Comments for Josh the Outspoken

Comment on Gazelle/upb status by Rehno Lindeque

Posted: February 13th, 2011, 11:26pm MST by Rehno Lindeque

That’s great! I’m looking forward to seeing progress on Gazelle again
Comment on Gazelle/upb status by Syl

Posted: February 3rd, 2011, 11:37am MST by Syl

agreed
Comment on Gazelle/upb status by Phil

Posted: January 28th, 2011, 3:39pm MST by Phil

This is exciting.

2 Star

Gazelle/upb status

Posted: January 28th, 2011, 11:35am MST by Josh
It has been just over a year since I last posted, leading some people to rightfully wonder whether my projects Gazelle and upb are abandoned. The answer to that question is a resounding “no.” I am more motivated to complete Gazelle and upb than I have ever been, and I have been working on upb actively lately (here are my recent commits).

However, if I’ve learned one thing over the last several years of working on Gazelle and upb, it’s that I am extraordinarily bad at knowing how close I am to being ready to release. I don’t want to make any predictions or promises. I honestly thought I was almost ready to release upb a year and a half ago, but I’ve almost completely rewritten it twice since then. Each time it gets significantly better and closer to being what I want it to be. Once I have a release I’ll describe more about the stages of evolution it went through and how each iteration was objectively better than the one before it.

The core interfaces have gotten to a point where I’m really happy with them and feel no more need to rework them. I think they will continue to evolve incrementally, but not in a way that requires redoing them completely. Here are the most core interfaces — if you’re interested in upb, I recommend reading these headers. I’ve just added substantial comments to explain them, and more than anything these will give you a taste of what upb is all about:
- upb_stream.h, the streaming interfaces for doing SAX-like tree traversal of protobuf data, and abstractions of fread()/fwrite(). These are probably the most important interfaces in all of upb, since all of the encoders and decoders are based on them.
- upb_string.h, an immutable, length-delimited (instead of NULL-terminated), reference-counted string type.
- upb_def.h, the data structures for a protobuf schema (.proto file) and routines for loading them.
By the way, I also mean to write zero-overhead C++ wrappers around the above to give you C++ programmers a nicer interface at no cost.

With those set, I have been rapidly getting everything building/working again. It’s a bit annoying to rewrite upb_def.c for the third time (literally) but it feels good knowing the interfaces are right.

So I have renewed optimism that I’ll be releasing soon. And once I’m happy with upb, it’s back to Gazelle.

Planet Matt

Bookmarks: Where to shop in Seattle - Page 3 - Style Forum

Posted: January 20th, 2011, 12:45pm MST

Includes a map of selected menswear stores downtown.
Bookmarks: Where to shop in Seattle - Page 3 - Style Forum

Posted: January 20th, 2011, 12:45pm MST

Includes a map of selected menswear stores downtown.
Photos: Chasing seagulls

Posted: January 19th, 2011, 7:33pm MST
Bookmarks: Is Parallel Programming Hard, And, If So, What Can You Do About It?

Posted: January 16th, 2011, 1:03am MST

Open-source book edited by researcher and Linux kernel developer Paul E. McKenney.
Bookmarks: Is Parallel Programming Hard, And, If So, What Can You Do About It?

Posted: January 16th, 2011, 1:03am MST

Open-source book edited by researcher and Linux kernel developer Paul E. McKenney.
Bookmarks: Shortchanging Your Business with User-Hostile Platforms

Posted: January 15th, 2011, 10:51pm MST

"AIR apps exist in an uncanny valley between a web application and a desktop application, and the result is unsettling and annoying." [I've run into this problem when writing apps that couldn't decide between desktop and web UI conventions.]
Buzz: Buzz by Matt Brubeck from Buzz

Posted: January 7th, 2011, 4:31pm MST

Enclosure: [download]

All you folks who like last-minute hacking, today's the last day to submit an entry to the Firefox Mobile Add-on Cup and win great prizes... [extendfirefox.com]
Bookmarks: Robot Driving Lessons – Michael Verdi

Posted: December 27th, 2010, 12:39pm MST

Robot Beltzner teaching Robot Madhava how to park, at the Mozilla office.
Bookmarks: Delicious → Pinboard username mapper

Posted: December 27th, 2010, 10:42am MST

Find your del.icio.us network on pinboard.in.
Bookmarks: Ben Morris: Intro to Scotch

Posted: December 24th, 2010, 6:06pm MST

"The goal is to create a functional programming language that's easy to use to quickly throw something together with, with highly readable syntax a la Python."
Bookmarks: Yacc is dead (Matthew Might, David Darais)

Posted: December 14th, 2010, 12:45am MST

"The payoff of these techniques is a small (less than 250 lines of code), easy-to-implement parsing library capable of parsing arbitrary context-free grammars into lazy parse forests. Implementations for both Scala and Haskell are provided."
Bookmarks: How to Become an Airplane Pilot

Posted: December 8th, 2010, 11:08am MST

Bookmarks: How to Become an Airplane Pilot
Weblog: Mobile web developers: Your users hate it when you do this

Posted: November 19th, 2010, 8:15am MST
Mobile Firefox beta releases include a “Feedback” add-on (like the one in Firefox 4 beta for desktop), which lets users tell us what they think about the new browser. Based on a sample of feedback from mobile beta testers, the most common complaints are about:
1. Speed
2. Fitting text to the screen when zoomed in
3. Mobile vs. desktop versions of web sites
The first two are straightforward, though not necessarily easy. We’re always working on performance, and we have experimental text reflow code (currently available in the Easy Reading add-on). But the last item is more complicated…
Browser detection pitfalls
Web sites can read the User-Agent header sent by your browser to see what browser and OS you are using. Some sites use that information to decide whether to send a “full” version of a web page, or a version formatted for mobile devices.

This can go wrong in several ways. If your browser or device is new, or wasn’t tested when a site was developed, that site has no way of knowing whether it is “mobile.” Users may also change their User-Agent to work around content restrictions or access different media formats. And some sites make incorrect assumptions, like that all browsers with “Android” in their User-Agent string are based on WebKit.

Even when the browser is known, readers and publishers might not agree about whether the mobile or desktop version is better. Based on our feedback, some users want to switch from full sites to mobile sites while others want just the opposite. And some devices, like large touch-screen tablets, combine aspects of handheld and desktop computers.
Solutions
Looking through these complaints, many people are under the mistaken impression that the browser, rather than the web site, decides whether to display mobile-formatted pages. Even the New York Times' David Pogue gets this wrong in his Galaxy Tab review:

When you visit sites like nytimes.com, CNBC.com and Amazon.com, the Galaxy’s browser shows the stripped-down, mobile versions of those sites. According to Samsung, there’s no way to turn that feature off and no way to visit the full-size sites. You can delete the little “m.” in the Web address until you’re blue in the browser, but the Galaxy always puts it right back.

Web developers: your readers are begging us to display your content in their preferred format. We want to help them, but we can’t do it alone.

(I wrote an add-on called Phony that lets mobile Firefox impersonate the User-Agent strings of other browsers. While this improves the experience on some sites, it breaks it on others. Masquerading as another browser can lead sites to serve non-standard markup that do not work in Firefox. Trying to solve this in the browser creates as many problems as it solves.)

Because browser detection is never perfect, web sites should let readers choose between mobile and full content. They can try to guess the right version by default, but please let users opt in or out.
Suggestions for web developers
Here are some first steps typical mobile web sites can take to make their readers happier:
- When appropriate, serve the same content to all browsers. You can use stylesheets and scripts to customize your layout for different display sizes, as in this beautiful site by Jon Hicks.
- There are valid reasons to use User-Agent sniffing. But if you must use it, test in as many browsers and devices as possible and learn the correct way to detect various browsers. For example, you can detect Gecko-based browsers by looking for Gecko and rv:, and you can detect mobile Firefox by looking for Fennec/.
- If a “mobile” user requests a page that isn’t available on your mobile site, don’t just redirect them to an unrelated mobile landing page.
- Let users switch from your mobile site to your full site and vice-versa. You can remember users' previous choices for convenience, but let them change their minds.
Further reading
For much more comprehensive development advice, see Yiibu’s thoughtful and practical approach to building sites that work across many different browsers and mobile devices.

One concern with the “same markup” approach is that it leads to heavyweight pages. Peter-Paul Koch explains how you can avoid sending unused images or markup to mobile devices by combinining CSS media queries and JavaScript to implement progressive enhancement.

Coming from a different perspective, Andrea Trasatti (former developer of the device-detection library WURFL) talks about problems in mobile User-Agent strings and how they could be more useful for device detection.
Photos: Box

Posted: November 5th, 2010, 12:16pm MDT
Bookmarks: W3C Web Events Working Group

Posted: November 5th, 2010, 9:14am MDT

"The W3C Web Events Working Group is chartered to develop specifications for physical multitouch interface events (including such related interface as pen-tablets, electronic whiteboards, and similar input devices), as well as for higher-level events which encapsulate touch interfaces, keyboard input, mouse control, and other input devices."
Bookmarks: Jigsaw Renaissance

Posted: November 4th, 2010, 11:39am MDT

"Jigsaw Renaissance is a learning and making community in Seattle, Washington." Located on First Hill.
Buzz: Buzz by Matt Brubeck from Buzz

Posted: November 1st, 2010, 5:36pm MDT

Do any of you Googlers know someone on the Reader team? Can you convince them to serve the iPhone/Android version ([www.google.com]) to Fennec users? It works much better than the desktop version.
Bookmarks: Steampunk and its discontents | MetaFilter

Posted: October 28th, 2010, 6:17pm MDT

Including a steampunk critique by Charles Stross
Buzz: Buzz by Matt Brubeck from Buzz

Posted: October 21st, 2010, 3:33pm MDT

Enclosure: [download]

"Yesterday at around 3PM PST, John McAllister began his quest to beat a 25 year-old "Joust" record of 107 million points. His game is being broadcast live."
Buzz: Buzz by Matt Brubeck from Buzz

Posted: October 7th, 2010, 10:31pm MDT

Enclosure: [download]

Yay! Our beta release is out. Give it a try if you have a high-end Android phone or a Nokia N900!
Weblog: What's different about Firefox for Android

Posted: October 4th, 2010, 5:00pm MDT
The Mozilla Mobile team has been working for several months on Firefox 4 for Android and Maemo (also known as “Fennec”). Here are some thoughts about the challenges we’ve discovered, how we’ve decided to solve them, and why Firefox is different from other mobile browsers.
Why
People often ask us why Android needs another web browser. These are a few things Firefox does that other Android browsers don’t:
- Syncs bookmarks, tabs, history, passwords, and form data to and from your phone. Firefox Sync and the Firefox Awesomebar help you enter URLs and passwords with less typing, and move seamlessly between your desktop and your mobile phone.
- Lets anyone write add-ons that can customize any part of the user interface. (Dolphin HD is another Android browser with some great add-ons, but its add-ons are provided by the browser vendor.)
- Uses the Jaegermonkey JIT, which is getting faster all the time. It runs JavaScript much faster than the Android 2.1 browser, and is even faster than the Android 2.2 browser on the WebKit’s own SunSpider benchmark.
- Supports web technologies like SVG, ECMAScript 5, WebM, and HTTP Strict Transport Security. Firefox for Android currently scores 217 points plus 9 bonus points on html5test.com. (Warning: Those tests can be deceptive; use them as a starting point for comparison only.)
There are several other browsers for Android, but all of them use the built-in WebKit rendering engine (except Opera Mini, which uses a proxy server for rendering). The same is true for iOS, which uses WebKit too – as do the latest versions of BlackBerry, Symbian, and Palm webOS. [Update, November 2010: Opera Mobile for Android is now in beta and uses its own rendering engine too.]

Part of the point of Firefox is to provide alternate capabilities, rather than reuse the built-in ones. Firefox for Android uses same Gecko engine as Firefox 4 for desktop. That’s how it can support features that Android’s WebKit doesn’t, like SVG and ES5. (Of course, WebKit supports some features Gecko doesn’t, which is why it’s great to have a choice.)
Speed and responsiveness
Early Firefox for Android builds were very slow compared to the stock browser. Performance is critical in a mobile browser, and our work in this area is starting to pay off. The new beta version is much speedier, and we have plans to make it even faster.

To make sure Firefox’s interface stays responsive even when pages are rendering, we split the browser into two processes: one for the user interface, and one to render and run scripts in web content. This was part of the Electrolysis project.

To make scrolling and zooming fast, Mozilla’s graphics team has implemented a new “Layers” architecture to allow hardware acceleration and other optimizations. Beta 1 is the first mobile Firefox release to take advantage of this work. Because of Electrolysis, we need to share layers between the UI process and the content process. These cross-process layers allow Firefox to scroll smoothly in response to user input, even if the content process is still busy rendering the page.

These are just the first steps toward making Firefox fast on mobile devices. Upcoming releases will feature OpenGL hardware-accelerated compositing, for improved scrolling of complex pages. They will also load web pages faster, thanks to optimizations of our inter-process communication.
Installation size: Problems and solutions
Not all mobile platforms allow browser apps to include low-level components like JIT compilers. Fortunately there are still platforms like Android, webOS, and Maemo that let apps bundle any libraries they want. But while Android allows us to distribute our own rendering engine and JavaScript compiler, it isn’t really built with apps like Firefox in mind.

Unlike browsers that use the stock WebKit library, Fennec must ship its own rendering engine. Many Android phones were built with just 64 MB to 512 MB of storage for apps. Users who think nothing of a 12 MB download to install Firefox or Chrome on a laptop may think twice before installing it on these phones! Storage space is much larger on newer phones, but this is still an issue for many users.

Even worse, a quirk of the Android NDK means these native libraries are saved twice – both compressed inside the APK and extracted to a folder for loading. For apps like Firefox that are mostly native code, this more than doubles the installation size. Other NDK apps like Google Earth pay the same double storage penalty.

To solve this problem, Mozilla’s Michael Wu wrote a custom dynamic linker that loads libraries from the APK without installing them to a folder. This cut the installed size by more than half, but increased startup time slightly. For newer phones with 1 GB or more of internal storage, we might want to let Firefox take more space but start faster. On phones with less storage, we can use the custom linker to save space.

Firefox 4 beta 1 needed about 40 MB of storage on Android. With the custom linker, Firefox 4 beta 2 takes a fraction of the space, and in Android 2.2 you can move almost all of it to SD.
Hardware compatibility
Stock libraries have another advantage: They can be optimized for specific hardware. In contrast, apps usually come in a single flavor for all devices. Firefox can use ARMv7 features like Thumb-2 and NEON to run as fast as possible on high-end Android phones – but with these optimizations it can’t run at all on low-end hardware. To run optimally on all current hardware, we’d need different builds for different devices. For now we’re focusing on the current high-end phones, which will likely be next year’s mainstream hardware.

Even in this smaller set of devices, we’ve run into problems. Most recently, we discovered that Firefox’s JIT code crashes unpredictably on Samsung Galaxy S phones. This seems to be a bug in the Android 2.1 kernel on these devices. Other developers are seeing similar problems, including crashes in the MonoDroid JIT and in Android’s own Dalvik JIT. For now, Firefox’s JIT features are disabled when running on Galaxy S hardware. This makes JavaScript slower, but a lot more stable. The problem is fixed in leaked Android 2.2 images, so we expect to re-enable the JITs before long.
Competition and choice
Firefox is built by Mozilla, a non-profit organization with a mission to promote openness, innovation, and opportunity on the web. We want our work on the mobile web to benefit everyone, not only Firefox users – just as Firefox on the desktop helped create a new era of innovation and standards for all browsers.

WebKit is an excellent project. But a growing number of mobile sites work only on WebKit. This is dangerously similar to the web ten years ago, when Internet Explorer had an overwhelming market share and many sites used IE-specific markup. That made it hard for other browsers to compete, which killed the incentive for the dominant browser to keep improving.

Upcoming platforms like MeeGo and Windows Phone will give WebKit some new mobile competition – but many users still can’t choose new browser technology without buying new hardware (and often new service contracts). We think you should have a meaningful choice of browsers on your current phone, just like you do on your computer. User choice will encourage all browsers to innovate and learn from each other, so they all improve faster.
Try it out
To check if your phone is compatible with Firefox 4 beta, go to our download page. This is a test release, and we aren’t finished fixing and optimizing it – but we are working hard. Let us know what you think!

We have a lot more work ahead of us. Our next releases will include even more exciting changes like the new Android skin, reduced installation size, and more speed improvements.

Comments for Josh the Outspoken

Comment on Gazelle/upb status and plans (aka: On Releasing) by Jonas Bülow

Posted: September 28th, 2010, 4:56am MDT by Jonas Bülow

Any updates? Dead project?

Planet Matt

Bookmarks: Race and ethnicity maps

Posted: September 20th, 2010, 5:59pm MDT

Census data on race, plotted on maps of major U.S. cities.
Weblog: Changes for add-ons in Fennec 2.0 alpha 1

Posted: September 2nd, 2010, 5:11pm MDT

Update (October 2010): We’ve changed the version number of the next mobile Firefox release from 2.0 to 4.0. The alpha release was numbered 2.0a1, but the first beta release will be numbered 4.0b1.

Last week we released a new alpha version of Firefox for Android and Maemo (a.k.a. Fennec). This release brings some major changes and new features for add-on authors. Our Fennec add-on documentation now has the details you need to start updating your Fennec add-ons or creating new ones.
What’s new for add-ons?
One very big change in Fennec 2.0 is Electrolysis, the project to move content and chrome into separate processes. Any add-on code that interacts with web content through the DOM must now be in a separate script that runs in the content process. For details, see the Electrolysis guide for add-on authors.

Fennec 2.0a1 also features new APIs for extending the context menu and site menu. See the User Interface Guide for links to documentation and example code.

The upcoming beta releases will include even more changes. Add-ons that use Fennec’s panning and zooming features will probably need significant changes for the new graphics code in Fennec 2.0b1. We will also include APIs for for add-ons to customize sharing and other new features. If you are working on an add-on that is affected by these changes, please let us know.
Get started
To start updating or creating your Fennec add-on, download Firefox for Android and Nokia N900 or get the emulator for Mac/Windows/Linux. When you’re ready, update your addons.mozilla.org listing and set the maxVersion to 2.0a1. Or you can start getting ready for beta by setting your maxVersion to 2.0b1pre and keeping up-to-date with our pre-beta nightly builds.

Josh

habes @ 2010-08-21T22:54:00

Posted: August 21st, 2010, 3:54pm MDT by Josh

Edit: See comments, I've changed my mind on this issue, but left my original writing here to show what my commenters are responding to.

This mosque debate keeps coming up, and shows no signs of going away, so I feel the need to express my opinion.

I'm somewhat distressed that my liberal friends who I usually see eye-to-eye with are almost all quite judgmental of the opinion that there could be any good reason to oppose this particular mosque on this particular site. There seems to be an overriding feeling that religious freedom demands that we not only allow this mosque to be built, but that we also give it our moral support.

My feeling at the moment is that there's not any good basis to actually deny the legal right of the mosque to be built at ground zero, but that I would criticize the imam's decision to move forward with the project at this location. I would join the voices that urge him to move it elsewhere. As Pope John Paul told the Catholic nuns who were going to move into a building near Auschwitz and build a cross there, "keep the idea, move the address."

If we agree that they have a legal right to proceed with the project (just as, by the way, their detractors have a legal right to protest outside it every day of its existence if they decide to) we can stop talking in terms of a religious freedom question and ask the more interesting question of whether it's a good idea that deserves our moral support.

The self-stated mission of the Cordoba Initiative is to improve Muslim-West relations. But this building hasn't even been built yet and already the project is worsening Muslim-West relations. To build bridges and cultivate trust you have to be sensitive to the other side. What if a Christian organization built a center like this in a Muslim country, but put a picture of Muhammad on the side? It would be an instant fail, even if their genuine intention was to improve relations.

I think the analogies are useful: an American cultural center at Hiroshima in 1954, or at Dresden? A German cultural center at Auschwitz in 1954, or on the beaches of Normandy? The analogies help because they take the "religious freedom" argument out of it. If we grant that the project is legal (thus satisfying religious freedom), it's becomes a question of whether the gesture is sensitive or not and whether we should give it moral support.

I don't think it's unreasonable to petition the imam to relocate this center. The US is almost 4 million square miles in area. New York City alone is 500 square miles. Why must this center be built on the 0.000025% of the country where Islamic extremists attacked us? Why would a center devoted to building east/west relations want to evoke that memory which is still quite fresh in people's minds? Why does the imam resist a land swap arrangement, which could be the perfect compromise -- allowing the center to be built without causing offense to >50% of Americans -- without any explanation but that the project will continue according to plan? I just find it hard to believe that someone who is already creating such division and mistrust is going to build east/west relations once the project is actually built.

Are there bigots who are opposing this just because they don't like Islam? Are there people who truly don't respect religious freedom? Probably. I haven't really read anything from people like that, but I'm sure they're out there. Don't let such people side-track the debate.

The point is that this project is achieving the opposite of its stated goal by uncompromisingly moving forward with a plan that is insensitive to the very Westerners it is trying to build trust with. What they're doing might be legal, but I think it's a bad idea and this is why I oppose it from a moral standpoint, though I don't deny that they have the right to build it.

Planet Matt

Bookmarks: Find the shortest route between two Wikipedia articles.

Posted: August 19th, 2010, 12:37pm MDT

Vice Admiral to ARM Architecture in four clicks!
Buzz: Buzz by Matt Brubeck from Buzz

Posted: August 15th, 2010, 2:12am MDT

I'll be in Mountain View this week (Mon-Fri) at the Mozilla office for a Fennec work week. Let me know if you're in the area and want to meet for lunch, dinner, whatever!
Journal: Home office away from home

Posted: July 21st, 2010, 6:43pm MDT

I've done working from home; I've done coworking; I've done the Mozilla Summit and soon I'll have my first work week. But this week I'm trying something different.

We're visiting my wife's parents in Port Townsend, WA for several days. Instead of taking vacation time, I'm working from their house this week. Since I already work remotely, my team will barely notice the difference. Meanwhile it will be a nice long visit for Sarah and her family, and I get to spend time with them in the mornings/evenings/mealtimes/weekend.

This has got me thinking about other new possibilities for nomadic working. It also seems relevant to this article on vacations and happiness research which says, "The most effective way to inoculate a vacationer against the deadening power of adaptation, however, may be the most counterintuitive — to break it up, to interrupt it with real life."
Photos: Summit at the Summit

Posted: July 10th, 2010, 12:25am MDT
Photos: Mascots

Posted: July 10th, 2010, 12:13am MDT
Buzz: Buzz by Matt Brubeck from Buzz

Posted: July 10th, 2010, 12:10am MDT

Enclosure: [download]

Paul Roget giving one of his most amazing demos of open web technologies and Firefox 4 features (SVG, CSS transitions, WebSockets, video, History API, accelerated graphics, and more) at the 2010 Mozilla Summit.
Bookmarks: rust

Posted: July 7th, 2010, 7:39pm MDT

"A very preliminary work in progress" programming language from Mozilla's Graydon Hoare (with help from Brendan Eich, Andreas Gal, Dave Herman, and more).

Looks like a very attractive multi-paradigm design. Lots of safety features (memory safe, no null pointers, objects are immutable by default, actors/tasks have no shared mutable state), structural typing, tagged disjoint unions, type inference, impressive static analysis features, native compilation (direct x86 codegen in Ocaml, and prototype LLVM backend), low-overhead C function calls and easy integration with gdb, valgrind, etc.

Build →

BuffaloFeeds » Friends

Feeds

Friends (100 unread)

Posted: January 25th, 2012, 9:15pm MST by Josh

Posted: January 25th, 2012, 8:33pm MST by Brian Slesinsky

Posted: January 25th, 2012, 9:28am MST by Josh

Posted: January 25th, 2012, 8:50am MST by Dave

Posted: January 21st, 2012, 12:07pm MST by Josh

Posted: January 2nd, 2012, 2:59am MST

Posted: January 2nd, 2012, 2:25am MST

Posted: January 2nd, 2012, 2:12am MST

Posted: December 9th, 2011, 1:33am MST

Posted: December 7th, 2011, 1:59pm MST

Posted: November 16th, 2011, 3:12am MST by Pierre Riteau

Posted: November 15th, 2011, 10:14pm MST

Posted: November 11th, 2011, 4:09pm MST

Posted: November 6th, 2011, 11:42pm MST

Posted: November 6th, 2011, 10:36pm MST

Posted: November 3rd, 2011, 10:35pm MDT

Posted: October 26th, 2011, 10:44pm MDT

Posted: October 24th, 2011, 10:10pm MDT

Posted: October 11th, 2011, 11:00am MDT

Posted: September 26th, 2011, 5:34pm MDT by Josh

Posted: September 17th, 2011, 4:08pm MDT by Josh

Posted: September 5th, 2011, 1:00pm MDT by Alexandre Abreu

Posted: August 26th, 2011, 6:47pm MDT by Josh

Posted: August 24th, 2011, 1:37pm MDT by Alexandre Abreu

Posted: August 5th, 2011, 2:20pm MDT

Posted: August 2nd, 2011, 6:46pm MDT

Posted: July 28th, 2011, 5:44am MDT by Rodrigo Moraes

Posted: July 27th, 2011, 5:06pm MDT

Posted: July 27th, 2011, 5:06pm MDT

Posted: July 26th, 2011, 12:10pm MDT

Posted: July 26th, 2011, 12:10pm MDT

Posted: July 20th, 2011, 7:50pm MDT by Rodrigo Moraes

Posted: July 15th, 2011, 5:56pm MDT

Posted: July 15th, 2011, 5:56pm MDT

Posted: June 27th, 2011, 9:56pm MDT

Posted: May 28th, 2011, 8:56am MDT by Peter Lund

Posted: May 16th, 2011, 5:04pm MDT by John "Z-Bo" Zabroski

Posted: May 9th, 2011, 12:01pm MDT by Josh

Posted: May 8th, 2011, 2:28pm MDT by Josh

Posted: May 7th, 2011, 11:30am MDT by Will

Posted: April 30th, 2011, 8:12am MDT by Brian Maher

Posted: April 29th, 2011, 4:25pm MDT by Josh

Posted: April 27th, 2011, 10:38pm MDT by Russ Cox

Posted: April 26th, 2011, 1:25pm MDT by Buffalo

Posted: April 26th, 2011, 1:02pm MDT by Phil

Posted: April 26th, 2011, 12:18am MDT by Josh

Posted: April 25th, 2011, 3:17pm MDT by Ricky

Posted: April 25th, 2011, 11:14am MDT by John "Z-Bo" Zabroski

Posted: April 24th, 2011, 12:37am MDT by Josh

Posted: April 18th, 2011, 11:36pm MDT by Josh

Posted: April 18th, 2011, 2:57am MDT by Josh

Posted: April 14th, 2011, 3:25pm MDT

Posted: April 14th, 2011, 3:25pm MDT

Posted: April 12th, 2011, 4:06pm MDT

Posted: April 12th, 2011, 4:06pm MDT

Posted: April 11th, 2011, 6:09pm MDT

Posted: April 11th, 2011, 6:09pm MDT

Posted: March 31st, 2011, 2:50am MDT by Josh

Posted: March 19th, 2011, 4:26pm MDT by Josh

Posted: March 4th, 2011, 12:58am MST by Josh

Posted: February 18th, 2011, 4:47pm MST

Posted: February 18th, 2011, 4:47pm MST

Posted: February 15th, 2011, 2:48pm MST