[p5.js 2.0 Bug Report]: Comparison algorithms may not work as expected

[pnngocdoan]

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

p5.js version

No response

Web browser and version

No response

Operating system

No response

Steps to reproduce this

I read through the code and believe that the visual comparison algorithm has three main parts:

• Use pixelmatch to compare pixel-by-pixel and highlight diffs in red
• Use BFS to find all diff clusters
• If 80% of a cluster has ≤2 neighbors, this cluster is considered a line shift and should be ignored.

I'm not sure if
(1) the args in pixelmatch are set properly to ignore diffs by aliasing (currently I saw that threshold is set at 0.5 and AA is set to false)
(2) the line shift criteria works well for curved shapes

Thus I modified the code a bit to print the diff images, with red as pixels caught by pixelmatch, green as pixels flagged as line shifts. Below is the code I used to test horizontal shifts (1px to 3px) and their diff outputs:

visualTest("with the default font", function (p5, screenshot) {
      p5.createCanvas(50, 50);
      p5.textSize(20);
      p5.textAlign(p5.LEFT, p5.TOP);
      p5.text("test", 0, 0);
      screenshot();
    });

visualTest("with the default font shifted 1", function (p5, screenshot) {
      p5.createCanvas(50, 50);
      p5.textSize(20);
      p5.textAlign(p5.LEFT, p5.TOP);
      p5.text("test", 1, 0); // shifted by 1px horizontally
      screenshot();
    });

visualTest("with the default font shifted 2", function (p5, screenshot) {
      p5.createCanvas(50, 50);
      p5.textSize(20);
      p5.textAlign(p5.LEFT, p5.TOP);
      p5.text("test", 2, 0); // shifted by 2px horizontally
      screenshot();
    });

visualTest("with the default font shifted 3", function (p5, screenshot) {
      p5.createCanvas(50, 50);
      p5.textSize(20);
      p5.textAlign(p5.LEFT, p5.TOP);
      p5.text("test", 3, 0); // shifted by 3px horizontally
      screenshot();
    });

when text shifts by 1px, pixelmatch can't catch that. I suspect the reason is that the threshold is set too high (at 0.5 out of 1), which increased the color tolerance. Although 1px shift is acceptable to us, I'm not sure this is what we are looking for out of pixelmatch (?) I'm assuming we want pixelmatch to catch all the diffs except AA ones and then let the line shift logic decide whether to ignore the diff clusters or not.

when text shifts by 2px, pixelmatch managed to catch most of the pixel diffs, however, the line shift detection didn't work for one cluster

when text shifts by 3px, pixelmatch managed to catch even more of the pixel diffs. However when I looked carefully, the lines don't actually "widen" (I expected the lines at 3px shift should look twice as thick compared to the 2px shift), instead the lines just connect more vertically, and still look like 1px strokes. If that's the case, I think the “t” shift should be flagged as a line shift, but it isn’t. Although I do believe a 3px shift should fail, but in this case it fails not because the diffs widen, but because the line shift logic didn’t catch and ignore them.

[limzykenneth]

This is a relatively tricky area as it is likely not possible to have precise pixel level comparison due to the way that text rendering differs between browsers and operating systems, which means we must approximate to a certain degree. The main thing we want to catch is obvious errors and rendering differing by 1 or 2 pixels should still be acceptable. I think one thing that we can potentially consider to reduce false negative is to make the test sketch larger so that it can draw larger text, that way the % threshold will represent a larger amount of pixels that is more indicative of visual errors.