How is raiasing task priority "requesting a bug to be fixed in a reasonable time scale"?

Let's not play on words, it's clear that priority reflects how important a task is. Do we know how many files are affected?

In T106516#1700723, @Nemo_bis wrote:

Let's not play on words, it's clear that priority reflects how important a task is. Do we know how many files are affected?

Is there even a way to query that?

It depends how you define "query". :)

$ curl -s 'https://upload.wikimedia.org/wikipedia/commons/archive/5/55/20151003120942%21Abraham_Lincoln_by_Alexander_Hesler.png' | exiftool -fast2 -
ExifTool Version Number         : 10.00
File Type                       : PNG
File Type Extension             : png
MIME Type                       : image/png
Image Width                     : 1208
Image Height                    : 1762
Bit Depth                       : 8
Color Type                      : Grayscale
Compression                     : Deflate/Inflate
Filter                          : Adaptive
Interlace                       : Noninterlaced
Background Color                : 255
Pixels Per Unit X               : 2835
Pixels Per Unit Y               : 2835
Pixel Units                     : meters
Modify Date                     : 2015:10:03 12:03:44
Image Size                      : 1208x1762
Megapixels                      : 2.1

Are all grayscale PNGs affected?

MariaDB [commonswiki_p]> select img_metadata from image where img_name = 'Abraham_Lincoln_by_Alexander_Hesler.png';
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| img_metadata                                                                                                                                                             |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| a:6:{s:10:"frameCount";i:0;s:9:"loopCount";i:1;s:8:"duration";d:0;s:8:"bitDepth";i:8;s:9:"colorType";s:10:"truecolour";s:8:"metadata";a:1:{s:15:"_MW_PNG_VERSION";i:1;}} |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Maybe:

MariaDB [commonswiki_p]> select count(img_sha1) from image where img_metadata LIKE '%s:8:"bitDepth";i:8;s:9:"colorType";s:10%';
+-----------------+
| count(img_sha1) |
+-----------------+
|          316217 |
+-----------------+
1 row in set (4 min 54.69 sec)

In T106516#1700699, @Yann wrote:

@Aklapper I hope that being a developer is not a new standard to request a bug to be fixed in a reasonable time scale?

@Yann: If you'd like to request fixing in some time scale, please add a comment suggesting the change and convincing reasons. Thank you!

Again here: https://commons.wikimedia.org/wiki/Commons:Village_pump#Increasing_amount_of_difficulties_at_Commons_Wikimedia

@Aklapper This is a serious issue. It affects a lot of files, and everybody including readers and contributors. This seems sufficient to me for this bug to be fixed quite soon.

So what exactly is needed to fix this? Either upgrade IM to 6.9.1+ or merge Bawolff's patch?

In T106516#1733802, @Tgr wrote:

So what exactly is needed to fix this? Either upgrade IM to 6.9.1+ or merge Bawolff's patch?

Is there any harm in upgrading? 6.9.0 is a fairly old version dating back to March.

Wikimedia's servers run Ubuntu and Debian; Ubuntu ships/provides IM version 6.7.7. Hence there is no harm but there are maintenance costs.

Hmm. That version dates back to March 2014. Is there a way to ask the ubuntu packagers to upgrade their package?

In T106516#1734326, @zhuyifei1999 wrote:

Hmm. That version dates back to March 2014. Is there a way to ask the ubuntu packagers to upgrade their package?

That's not how Ubuntu works. If we cared about always having recent software, we'd run a different distro. :)

https://commons.wikimedia.org/wiki/Commons:Village_pump#Images_rendering_darker_in_the_thumbnails

Change 226665 abandoned by Brian Wolff:
Workaround im bug where greyscale have assumed gamma of 1

Reason:
Abandoning this for now. I'm not going to work on this in the near-future. Someone else will have to take it up.

https://gerrit.wikimedia.org/r/226665

While we're waiting for someone else to take this up, a workaround on a per-image basis is to download the image, convert it to grayscale using a more recent version of Imagemagick on your local computer (using "convert {input_image} -type GrayScale -depth 8 gray.png", and re-uploading it. It would be nice, however, if this were fixed on the server somehow.

@Aklapper @brion surely it is reasonable that we get this fixed. Some of us just use plain old image editors and trying to install software and running command line fixes like this just seems unreasonable.

upload version https://upload.wikimedia.org/wikipedia/commons/1/12/Lady_Peel.png
thumbnail version https://commons.wikimedia.org/wiki/File:Lady_Peel.png (pale and washed out)

Eight months later it seems reasonable that it gets fixed, quality is important when reproducing images.

Would you volunteer to compile and maintain a custom package to deploy on servers?

This doesnt need a custom compiled version of image magick. If someone wants to address the code review comments on the abandoned patch, that will fix it. I just dont have plans to work on that patch in near future, so I abandoned it.

I volunteer to do lots of things and have for years. So please don't take
that path.

I know my capabilities, and this isn't one of them. So that is neither a
reasonable nor practical solution. Not sure that it is even valuable as
rhetoric, and it wasn't helpful.

I also know that grey scale pngs are not unexpected especially when
reproducing 19th century printed works. I am also told that these images
shouldn't be reproduced as lossy jpgs. So do nothing may be practical, but
not clearly reasonable.

While for "new" pngs, there is a solution for the Linux technorati which is
reasonable, it is not practical for the vast bulk of users. It isn't
practical for all the existing images.

If someone was to set up a bot to hunt and convert existing and new
"problematic" images, that is reasonable and maybe practical for someone.
It may be a reasonable first aid if there is no system fix, but it is
reactive and not a fix of the root cause.

After eight months to still have "do nothing but stare at it" as the
solution is an issue. So maybe we have to talk about a less than perfect
solution of what is possible, rather than do nothing.

End users have no control, whereas you developers have complete control of
the resouces and the solution. So please can we look for a solution and if
that means a custom build for a while let the tech heads have that
discussion. Or that the organisation uses its partnership relations to have
the issue fixed at source. (Clearly some politics there that I don't
understand.)

@Storkk your last assertion isn't quite true, FYI

I removed my comment in order to attempt to avoid derailing this bug report, which I have failed to do. Apologies to all for that.

I am happy that @Bawolff corrected me in T106516#2073472.
So anybody strongly interested in seeing this fixed will have to convince a developer to improve https://gerrit.wikimedia.org/r/226665 .

@Bawolff I'm still a bit swamped with recent non-code stuff but I'll make sure it's in my review queue, will try to take a look over it soon. If anybody else gets to it sooner, awesome. :)

Change 226665 restored by Brion VIBBER:
Workaround im bug where greyscale have assumed gamma of 1

Reason:
(reopening to switch ownership and rebase)

https://gerrit.wikimedia.org/r/226665

I guess this is the only task that has a lot of "The World is Burning" token...

In T106516#2076172, @brion wrote:

I'm still a bit swamped with recent non-code stuff but I'll make sure it's in my review queue, will try to take a look over it soon.

@brion: Any luck with that? :) https://gerrit.wikimedia.org/r/#/c/226665/

Are we any closer to getting a permanent resolution to this?

I have just uploaded a batch of images like

https://commons.wikimedia.org/wiki/File:An_anxious_time_near_goal._%22Shoot!%22.png
(pale and washed out) compared to the upload
https://upload.wikimedia.org/wikipedia/commons/c/cc/An_anxious_time_near_goal._%22Shoot%21%22.png

and for me the problem still exists. I am using GIMP, so even if someone can tell me what I should do differently with GIMP. Thanks.

I have interrogated GIMP, and I will now export from GIMP forcing the gamma setting (which GIMP indicates shouldn't need to be set)

In T106516#2625315, @Billinghurst wrote:

I have interrogated GIMP, and I will now export from GIMP forcing the gamma setting

Interesting. Let's document this workaround somewhere. Maybe https://commons.wikimedia.org/wiki/Commons:File_types#PNG ? Or at least the task description.

Is there still no way of a permanent fix? I have been using GIMP with forced gamma settings for a while and even reuploaded lots of my older images for this reason. But even that didn't work with some of my recent uploads, e.g. https://commons.wikimedia.org/wiki/File:Andr%C3%A9_Campra_-_Idom%C3%A9n%C3%A9e_-_title_page_of_the_score_-_Paris_1712.png - the jpg version of that image looks much better in the overview: https://commons.wikimedia.org/wiki/Category:Idom%C3%A9n%C3%A9e .

https://commons.wikimedia.org/wiki/File:Esperanza_Barrios.png is another case reported to IRC a few minutes ago.

This problem is not just GIMP-related. I regularly make PNG files with the FastStone Image Viewer - typically from converted jp2-files - and it irritates me considerably that they get such a pale appearance in the thumbs. What can be done?

In T106516#3762566, @Rsteen wrote:

This problem is not just GIMP-related. I regularly make PNG files with the FastStone Image Viewer - typically from converted jp2-files - and it irritates me considerably that they get such a pale appearance in the thumbs. What can be done?

It has never said to be a gimp issue, it is a known Wikimedia issue. That said there is a means for GIMP to export PNG files that do not suffer the identified issue. I would hazard a guess that your software has a solution too. See if there is a means to force the gamma setting.

Current possible example: https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(technical)&oldid=859512933#Thumbnail_too_dark?

AFAICT production is using the expected, "normal" version of ImageMagick for our variant on Debian Stretch: 6.9.7-4.

Did the patch that fixes this in 6.9.0-1 get released into 6.9.7-4? If so, that means that any new thumbnails are being created OK but the old cached thumbs are wrong. So to fix this we'd need to delete the wrong ones and have the image scalers re-generate them. To avoid a stampede, SRE would need to do something special to identify which to delete and iterate slowly. Not sure if such code exists? (Fastest option)

If not, someone could ask the Debian ImageMagick maintainers to back-port it into their custom release for Debian Stretch, at which point production would get eventually get upgraded (and the file regeneration above would then have to happen). (Medium option)

If not, we'd need to move Wikimedia production's version of ImageMagick away from the widely-tested and security-supported upstream version with a custom backport, which is generally a Bad Idea™. I don't know if SRE have the bandwidth to do that? (Slow, hardest option)

If not that, we'd need to wait for Wikimedia production to move to Debian Buster. I don't imagine that's going to happen until early 2020 at the earliest; there's not even a planning ticket for that yet — it's not expected for Debian to declare it a stable release until mid-2019. (Slowest option)

Change 226665 abandoned by Brian Wolff:
Workaround im bug where greyscale have assumed gamma of 1

Reason:
I'm not working on this anymore

https://gerrit.wikimedia.org/r/226665

Thumbnails on Wikimedia sites are now handled by Thumbor, not the MediaWiki image scaling system. Thumbor is also on Debian Stretch and using the provided imagemagick 6.9.7.4.
The original versions of https://commons.wikimedia.org/wiki/File:The_Residence_of_the_Count_of_Wassenaar,_Lord_of_Obdam,_Kneuterdijk,_The_Hague.png and https://commons.wikimedia.org/wiki/File:Gilbert_Burnet,_engraved_by_John_Rogers_after_John_Riley.png appear to be rendering correctly.

T106516#3626359 appears to be more about the lack of conditional sharpening in PNG thumbnails. That's irrelevant to this task and is tracked in T192744: PNG thumbnail looks significantly blurrier than JPG thumbnail.

https://commons.wikimedia.org/wiki/File:Esperanza_Barrios.png is very slightly too dark still, and https://commons.wikimedia.org/wiki/File:Bell_Relay_Computer.png is very dark. Cached older versions of the second file are slightly darker than newly-generated versions, but are still too dark. Testing locally with ImageMagick 7.0.10-11 produces an acceptable output.

old 768px version, last-modified Fri, 14 Sep 2018 21:28:37 GMT

Phabricator's thumbnailer applies ImageMagick to all three, making them look the same unless you view the original files.

I don't know if this will be fixed in Buster with ImageMagick 6.9.10. It's worth testing at least though.

I've just noticed that Bell_Relay_Computer.jpg uses

Colorspace: LinearGray
Type: Grayscale

The other files use

Colorspace: Gray
Type: Grayscale

so this is probably a different, but related, ImageMagick bug.

File:Bell Relay Computer.png has a size of 10.6 MP, which puts it over the 10 MP minimum Vips scaling size. That means it is first scaled by Vips. The file also does have a gAMA chunk, and the gamma value is set to 1. As I previously noted, it's also in the LinearGray colorspace. When Vips scales the image, it sets the gamma value to 0.454545 and sets the colorspace to Gray, but doesn't transform the values correctly.