Bartender Test Reports

The following reports were generated using Bartender (Browser Automated Render Testing Driver)

The testing below was performed using Mozilla Firefox 68.0, Google Chrome 75.0.3770.142, and Servo (git hash 9451a00, issue 23905). Bartender was configured to run each test with a maximum of 25 trial iterations per run and a maximum generator size parameter of 50. A rendering difference found during a test run is shrunk to more minimal test case that still induces a rendering difference. The shrunk test cases are the ones reported below. All the tests were run on a 8 CPU 3.2GHz Intel i7 system running Ubuntu 16.04.

Firefox compared to Chrome

This data was generated from 5000 test runs comparing Firefox and Chrome. Out of the 5000 test runs, 1805 runs resulted in rendering differences, out of which 1695 are distinct test cases. In total 712,193 iterations (page loads) were performed (trial iterations + shrink iterations) over a period of 89 hours.

• Flat List of Rendering Differences
• Rendering Differences Arranged by Tags/Attributes and Tags/Properties

Servo compared against Firefox+Chrome

This data was generated from 1000 test runs comparing Servo against Firefox and Chrome. Out of the 1000 test runs, 257 runs resulted in rendering differences, all of which were distinct. In total 112,073 iterations (page loads) were performed (trial iterations + shrink iterations) over a period of 60 hours.

• Flat List of Rendering Differences
• Rendering Differences Arranged by Tags/Attributes and Tags/Properties

General Notes

Bartender uses several techniques to increase efficiency and minimize noise from normal browser rendering differences including:

• Using of generic CSS resets defined in /static/normalize.css
• Using Bartender specific hacks defined in /static/rend.css
• Using the Ahem font defined in /static/Ahem.ttf (font glyphs are solid rectangles)
• Wrapping each font glyph with a 'wrap-ahem' span to mitigate Servo issue #24042
• The Servo tests use HTML/CSS grammar weights that are resulted from the Firefox/Chrome tests. This reduces the probability of testing HTML and CSS elements that already caused differences between Firefox and Chrome.

To get the same rendering behavior you will need the font and CSS files listed above alongside the HTML from the test case.