What could be wrong?

Web compatibility over a coffee - notes, not slides

Notes from meetup in Stockholm

Today, we had a small meetup in Stockholm, called “web compatibility over a coffee”. As it was in a café without projectors and such, I didn’t prepare slides or a presentation - but I jotted down some notes, gathered some bugs to look at and some screenshots. This is basically the basis for our conversation

Why web compatibility?

  • One altruistic reason - improve the web
  • One selfish reason - improve the UX for browser users

The web is the most widely used programming environment. The world’s greatest experiment in the capabilities and limits of human logic. A giant test suite for web browsers. And it takes two to tango: a web browser and a web site. A poor user experience is bad for a browser, whether it technically is our fault or not!

Some history:

Tech Evang component was added to Bugzilla in 2001.

Right now 1381 open bugs, of those 412 ASSIGNED

Around 14 000 bugs accumulated in component in total

Tech evangelism issues graph over time

Sounds like a lot of work.. why does this happen?

Well, it is a lot of work - and there are many reasons we may want to contact a site..

Old technology:

Browsers grow unexpected features sometimes:

The certainty of change:

Pick your poison - browser-specific code:

User-Agent detection:

(Sadly, Mozilla is also doing it - we’re making things worse, nudging some selected partners to look for a magic “.1” token in the version number. At the meetup, this led to a very interesting discussion about possibilities for adapting content to a device by measuring things like load- and scroll-performance, simplifying or enhancing the page correspondingly. It would certainly be interesting to see some experiments here!)

Sheer confusion:

So, where’s the frontier now?

Definitely on Mobile. The -webkit- mess. The iOS / Android duopoly which is rendering-engine-wise a monopoly. Desktop is however making a bit of a comeback due to mixed content blocking and generally stalling market share of Firefox

Any community forming?

This sort of work is both suitable and not for community building.

  • Suitable because there’s a lot of work to do, and much of it is just testing and outreach (i.e. doesn’t need extreme technical skills).
  • Suitable because contributors can learn a lot from analysing code and studying the existing analysis.

  • Not suitable because contributors tend to just pay attention to “their own” sites, do one or two bugs, and disappear
  • Not suitable because outreach is hard, and there is often a long time to wait before you see results.

For a wider reach, we’ve built webcompat.com. It’s becoming a cross-browser effort - all browser vendors struggle with old code, we all want the web to move forward. This helps but we still need more engaged users - and users who are prepared to stick around long-term, and work on issues across several countries, rather than just doing a few bugs or some outreach in a specific country - valuable though it is. There’s also arewecompatibleyet.com which is more Gecko-centric and presents Bugzilla Tech Evangelism data in a different way.

Can we automate any of this work?

Sure. You might think of simply doing wget / curl and diff commands. You’re wrong, it’s much harder than that.. Here are some of my site tests which I run with SlimerJS. We’re also working on using a volunteer-developed distributed infrastructure which can run tests continuously from several locations across the world. It will also support heuristics for discovering compatibility problems. This might take care of some of the boring work (testing to find problems, analysing simple issues) - and then we and the volunteers will handle the problems that require humans.

Thank you, Soumya, for organising the meetup!

CNN video redirects

Here are snippets from the JavaScript behind bug 970849, freshly served by CNN’s website:

var index='/index.html',

Hm, no Mobi there? What about a Tablet? Or even better - since this is client-side already - drop the names altogether and look at width and height to figure out how to adapt the content? The problem with name sniffing is that you’ll only be adapting to the few devices you already know the names of, not the millions of future devices you don’t. So, no surprise that this detection fails to handle Firefox OS smartphones, for example. Not to mention the Opera Minis and Mobiles, the phones shipping Access’s browsers etc.

device=String(((match instanceof Array)?match[0]:match)).toLowerCase(),

That instanceof call is a good idea though - as long as you know you might end up with device being “null”.


So, want to bet that this redirMap is going to contain rules for the “null” device?

'iphone': function(){

Snipping away some complicated rules for the iPhone..


..and even more rules for the iPad.

return href.replace(path,path.replace(index,'/')+"standard.html");

This statement handles redirects for people who type cnn.com/video/ into those specific browsers this script cares about. It also does a special replace for people who type cnn.com/video/index.html. But what about users who type cnn.com/video without the final slash? Well, they don’t get to see any video.. Here’s what they get:

CNN videostandard.html 404 page screenshot

Onwards through the rest of the JavaScript:


Oh, cute - there’s a special redirect for Google TVs. Somebody from the higher echelons of Google must be a Turner corporation board member or something..


And where is our “null” device handled, you might ask? Don’t be silly - being forward looking is so over-rated.

if(device in redirMap) document.location=redirMap[device]();

So this part just runs the hand-picked redirect rules based on the device you’re using (if they think it’s worth redirecting for)..


And this confirms it’s all built on top of the much abused navigator.userAgent property. As if we didn’t guess that from the first .match().

So, it’s official: iPhoneDroidKindle-level rich people who remember to end URLs with a slash is the only demographic the CNN cares about.

Yeah, what’s the point of a big world when you have a small mind?

IE girl

2014-09-23 / Site compat, Found code

SVG Girl website screenshot

Once upon a time, Microsoft wanted to promote IE9. I guess the IE6 stigma still stuck (9 is just a 6 upside down after all), so they were looking for a way to sprinkle magic coolness over the release. Enter SVG Girl - because what could sprinkle more cool than some anime with random strings of katakana? So there it is, a smooth, flashy and ultimately extremely odd animation of a girl who gets IE9 on her phone and gets … possessed. I can’t think of any other way to interpret that story - IE9 seems to enter her like an evil spirit and take control of her..

IE9 is history, but when you ship code on the web it tends to stay there. So this poor possessed girl is still out there promoting IE9. She’s even critical of visitors using later versions of IE - forever telling them that IE9 is the greatest thing ever. Hey - did IE9 even run on any phone in the Japanese market?

In any case, the page hangs in Firefox and Chrome. The simple reason is explained in bug 701672: when an element has id=myElm, window.myElm will typically refer to the element. IE turns out to be inconsistent for IFRAMEs, and window.myElm refers not to the IFRAME element, but to the window inside. It probably seemed like a good idea at some point..

The SVG Girl’s JavaScript expects IE’s implementation - or it expects other browsers to stick to their guns and not create references to tags with @id attributes on the global object.

Makes you wonder - was the coder possessed by IE9 too?

View test code on arewecompatibleyet.com

While the site arewecompatibleyet.com ships with test results from my web testing, it’s until now been somewhat of a mystery how those tests actually work.

Now, even casual visitors who do not want to search through big files of code on GitHub can get a sense of how the tests work. If you scroll down to the bottom of arewecompatibleyet.com’s front page, you will find a section listing all tests that output something which doesn’t match the state of the bug. If a bug is closed as fixed and the test fails, or the bug is open but the test passes, it’s listed.

There’s a tiny new feature: a link labelled “View test code” for each entry in this table. It brings up a small box where you can not only view the test code, but also copy a chunk of JavaScript that can run from the console.

For example, there’s an apparent regression on thestar.com, regarding bug 842184, no mobile content on m.thestar.com for Firefox on Android. Here’s the new feature at work, showing the test code (click to enlarge):

screenshot of AWCY site showing test code for bug 842184, regarding thestar.com site

Clicking in the grey area will select it for copying. The code you copy includes some “scaffolding” to make the test actually run - the code that is written for bug 842184 is the part inside the steps array:

    var i=1, steps = [

function (){return mobileLinkOrScriptUrl() /*(regression test, expected to pass)*/ && hasViewportMeta() /*(regression test, expected to pass)*/}

    function doStep(){ if(typeof hasViewportMeta !== "function"){var s=document.body.appendChild(document.createElement('script')); s.src='http://hallvord.com/temp/moz/stdTests.js'; s.onload = doStep; return;};if(steps.length){var result = steps[0](); console.log('test step '+i+' says: '+result); if(result !=='delay-and-retry')steps.shift(); i++; setTimeout(doStep,300);}} doStep();})()

The instructions say you should switch User-Agent (either with a suitable extension or through about:config) to spoof Firefox OS (that’s actually a data bug since we’re dealing with a bug for Firefox on Android here) and load m.thestar.com. Clicking the site link opens it in a new window.

When the site loads, we open developer tools, go to the console and paste the testing code:

screenshot of console after pasting the test code - we see the output is false

As we see, the code returns false (aka ‘not good’). Now we can play around with the test code - for example by removing the code calling mobileLinkOrScriptUrl() which I highlighted in the screenshot above. And voila:

screenshot of console after pasting the edited test code - we see the output is true

So it looks like the test should no longer look for the ‘mobile’ keyword in file names. Unfortunately, testing the code we end up with against the desktop version of the site also returns “true”. To fix this test we should dig a bit deeper and look for some difference in the DOM. It’s sometimes hard to find stable features that are different - it can be an interesting challenge. Have fun if you try to help!

Testing live websites at scale

2014-06-12 / Mozilla, Site compat, Testing

When I started working at Mozilla, one of the things I wanted to attempt was live testing against real sites - at scale. One year on, I have a sort-of-good-enough framework, over 800 tests, thousands of test results from monthlyish test runs, and a much better infrastructure around the corner thanks to the work of volunteer Seif Lotfy. It’s probably a good time to document how I do my monthlyish test cycle.

I start by loading arewecompatibleyet.com, nicknamed AWCY. This site tracks Tech Evangelism bugs relating to specific websites, and associated test results.

The site has a secret feature I run from the console: a method to list all the Tech Evangelism bugs that it does not have test results for. So my first step is to open the console and type


Screenshot of the Firefox console showing the listMissingTestBugs method and its output

I copy the tab-separated output data to a text editor and save it as sites.txt in a brand new folder named missing-2014-07-12.

(Well, actually - that’s not true. First I paste them into the first textarea on sitedata-explorer.htm to remove any entries we have tests for already - sometimes test exist but for some reason failed to output any test results during the last test run. This means AWCY forgets that the test exists, and we risk creating a duplicate test. The output from this script does into sites.txt. I could make this step superfluous by changing the next script to check for existing tests..)

Now I’m going to run a script for playing through the URLs in a real Firefox instance, using the Mozilla Marionette automation framework. This script will attempt to generate automatic tests, and take screenshots which I can review to verify the generated tests. (In the not-too-distant future this script will retire and be replaced by Compatipede which extracts even more data points of information, enabling more varied and hopefully robust tests.)

First I run the Firefox instance I want to control, making sure I pass the -marionette argument for enabling Marionette and the -no-remote argument so it doesn’t just hook up with my existing Firefox instance:

c:\Program Files (x86)\Nightly\firefox.exe -marionette -no-remote -p tester2

Having pointed testsites.py to the right directory by editing the dirname variable, I can simply do

python testsites.py

and it should run through all the data in sites.txt.

Screenshot of the Marionette script running, with Firefox being controlled by the script

It outputs some statistics from the tests while running. The Mozilla Marionette framework (or Gecko itself) tends to hang on certain points during this process, maybe it is not always firing the right events when a site stops loading? Hence this requires a bit of babysitting - if it hangs, I’ll have to resume from a specific index in the list:

python testsites.py -s 12

However, here there is a small wart in the process. The testsites.py script will generate several files: screenshots, comparison screenshots showing the site with two different user-agent strings, and sitedata-automated.js full of suggested automated tests. This file is a JavaScript file with function definitions and thus not valid JSON, and testsites.py can’t currently read it. To avoid overwriting the existing data when we resume running testsites.py, I have to review those tests or stash them somewhere before resuming.

Reviewing the tests is a crucial step - I verify that the user-agent and test code looks sensible. For example, the script suggests this test:

"1003466": {
    "url": "http://www.sismologia.cl", 
    "steps": [
        function(){return hasViewportMeta() && location.hostname === "" && mobileLinkOrScriptUrl();}
    "ua": "FirefoxOS", 
    "title": "sismologia.cl sends desktop site to Firefox OS"

This looks strange - why would mobile browser be taken to an IP-address? This might be a temporary solution while the mobile site gets a proper hostname. I might decide to remove the hostname test and verify that the mobile site has META viewport and the desktop site doesn’t.. Or when the script proposes this test:

"1019204": {
    "url": "http://match.com", 
    "steps": [
        function(){return hasViewportMeta() && location.hostname === "touch.uk.match.com" && hasHandheldFriendlyMeta();}
    "ua": "FirefoxOS", 
    "title": "match.com sends simplified site to Firefox OS"

it will probably not run very well on the distributed, international infrastructure - not all machines will be in a location that match.com redirects to their UK site. This is probably a better version:

"1019204": {
    "url": "http://match.com", 
    "steps": [
        function(){return hasViewportMeta() && location.hostname.indexOf("touch.") === 0 && hasHandheldFriendlyMeta();}
    "ua": "FirefoxOS", 
    "title": "match.com sends simplified site to Firefox OS"

Some bugs are not testable. For example, in this test run AWCY included bug 895485 in the list of bugs that require a test. This is more of a meta bug, it doesn’t really describe a specific issue on a specific site, and hence we don’t want a test for it. I’ve added a file called ignored_bugs.txt which lists the bugs testsites.py will ignore and not attempt to generate tests for. Each bug number has a comment that explains why we want to ignore that bug. Let’s add 895485.

When this test run is done, when all tests in sitedata-automated.py are reviewed and copied to sitedata.js, I will also go through the generated sitedata-missing.js file. The script has prepared empty JSON blocks for all sites it could not generate tests for. For example

"1019380": {
    "url": "http://www.hotels.com/", 
    "ua": "FirefoxOS", 
    "steps": [
        "function(){return }"
    "title": "hotels.com sends Desktop Content to Firefox OS and Firefox Android"

(First thing to do is to remove the quotes around the function - this file is valid JSON.. apologies for not hooking up a JS parser here. I do this only once a month and the script is scheduled for retirement anyway..)

Investigating, I see there’s a good reason why the automation could not generate a test: the problem seems fixed! Yay! Last update on the bug is only 10 days ago, and we haven’t even contacted them - but it works fine now. So testsites.py obviously did not detect any problems, but we can write a test manually and keep it around to detect regressions:

"1019380": {
    "url": "http://www.hotels.com/", 
    "ua": "FirefoxOS", 
    "steps": [
        function(){return location.pathname.indexOf('/mobile') === 0}
    "title": "hotels.com sends Desktop Content to Firefox OS and Firefox Android"

So this test will keep running - if they tweak their browser detection again, and no longer detect Firefox OS correctly, the test will flag a failure and announce it on the AWCY site.

Finally, when the new tests are added, I’ll do a test run and add test results to the AWCY site. I use a SlimerJS-based test script. I can run either individual tests:

slimerjs -P SlimerJSTester slimertester.js 1019380 1019204

or just run them all:

slimerjs -P SlimerJSTester slimertester.js

It takes quite a while to run 8-900 tests - but will eventually output results-2014-07-12.csv, and I’ll add that file to AWCY’s data/testing directory and update index.json to add a reference to the new file. And then the most exciting step: reloading the AWCY site to check the brand new list of fixes or regressions at the bottom of the front page..

The process still has some warts, but it works fairly well. We’re testing the web at scale, and we’re ready to scale even further. If you want to add or improve tests, pull requests for sitedata.js are naturally welcome!

The tests and infrastructure are not intended to be limited to Gecko/Firefox issues. With the growing, cross-browser bug list of issues on webcompat.com, now is a good time to start adding tests for other browsers’ issues to our framework. Seif’s “Compatipede” infrastructure will make cross-browser compatibility testing at scale even more powerful - we’ll run both “exploratory” tests for various configurations and reviewed, bug-specific tests. We’re in for a fun ride while scaling up!

(Edit september 2014: removed the link to the sitedata-missing file, it’s now outdated.)