If at all possible, can I get the train window at 14:00 local time (CET) which would be 13:00 UTC. That will be easier to manage for me.

In T263184#6606955, @hashar wrote:

If at all possible, can I get the train window at 14:00 local time (CET) which would be 13:00 UTC. That will be easier to manage for me.

Looks like I had this task assigned incorrectly vs our weekly meeting notes. You're backup for this week, but you'll be primary on T263186: 1.36.0-wmf.20 deployment blockers

Consider this train blocked until the cause behind T267668 is either fixed or reverted in master. I've prepared a revert at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/640503, but giving my team until the branch cut next Monday/Tuesday to try to fix it in-master.

We have rolled out the fix for T267668: Some recent Commons uploads not available on other wikis (2020-11) yesterday (Nov. 12th). It hasn't been rolled back in master as Timo said.

Change 641319 had a related patch set uploaded (by DannyS712; owner: trainbranchbot):
[mediawiki/core@wmf/1.36.0-wmf.18] Branch commit for wmf/1.36.0-wmf.18

https://gerrit.wikimedia.org/r/641319

Now fixed in master and included in wmf.18:

In T267668#6626361, @gerritbot wrote:

[mediawiki/core@master] filerepo: remove repo name from getSharedCacheKey()
https://gerrit.wikimedia.org/r/640868

In T267668#6626368, @gerritbot wrote:

[mediawiki/core@wmf/1.36.0-wmf.18] filerepo: remove repo name from getSharedCacheKey()
https://gerrit.wikimedia.org/r/641290

T268012 and T268008 are two GrowthExperiments issues that are not fixed in wmf.18 as far as I’m aware. Both have fixes available, and for the former I just deployed the fix for wmf.16, but I’m not sure how to backport changes for not-yet-deployed trains (last time I tried, it didn’t quite work out), so I’ve left the wmf.18 branch alone for now. @kostajh can tell you more about whether those issues have any serious user impact or “just” logspam; depending on that, I expect you’ll decide whether to apply the backports to wmf.18 before the initial rollout, or afterwards.

hi @dancy, we've backported a fix to wmf.16 but need https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/641294 also backported to wmf.18. I've put it on the deployment calendar for the upcoming evening backport window but that will happen after this gets to group0. Which, now that I'm writing this, I think is probably OK, but just a heads up about it.

@kostajh I will include https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/641294 in the group0 rollout.

Change 641319 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.18] Branch commit for wmf/1.36.0-wmf.18

https://gerrit.wikimedia.org/r/641319

@kostajh https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/641294 is deployed to group0.

I hope to see something to deal with the T268008 log messages soon. Thanks!

In T263184#6628840, @dancy wrote:

@kostajh https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/641294 is deployed to group0.

I hope to see something to deal with the T268008 log messages soon. Thanks!

@dancy the patch attached to that task would fix the problem, I think (cc @Tgr and @Catrope to review) but it seems like a bigger problem with the job queue infrastructure, not specific to that particular job.

In T263184#6628846, @kostajh wrote:

@dancy the patch attached to that task would fix the problem, I think (cc @Tgr and @Catrope to review) but it seems like a bigger problem with the job queue infrastructure, not specific to that particular job.

Indeed, the larger issue is very troubling (our job doesn't do anything important, but if the parameter corruption affects some job which writes data, and it gets corrupted in such a way that it doesn't trigger an error, just results in the wrong data being written, that would be rather bad) so I don't think we should suppress it until the core issue is found.

We found the underlying issue (it was specific to the extension) so the logspam will be fixed soon.

In T263184#6629462, @Tgr wrote:

We found the underlying issue (it was specific to the extension) so the logspam will be fixed soon.

Looks good now. Thanks!

FYI, am backporting https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventLogging/+/641768/ into wmf.18 now with help from Urbanecm.

T267668: Some recent Commons uploads not available on other wikis (2020-11) seems to be cropping up again.

In T263184#6634017, @Lucas_Werkmeister_WMDE wrote:

T267668: Some recent Commons uploads not available on other wikis (2020-11) seems to be cropping up again.

Thanks for the note. Is this a request to hold the train, or just a hint that there may be a patch coming in soon?

I’m not the right person to decide that, sorry – I just saw a few reports of the few issue and thought it was worth mentioning here.

The train has been rolled back to group0 (i.e., group0 = 1.36.0-wmf.18, group1 and group 2 = 1.36.0-wmf.16).

T267668 is now blocking the train.

@Krinkle deployed the fix for T267668 to wmf.16 and testing looks good.

group0: wmf.18
group1: wmf.16
group2: wmf.16

@greg says we can finish the wmf.18 train on Monday and Tuesday.

wmf.18 has been rolled back out to group1.

Current status:
group0: wmf.18
group1: wmf.18
group2: wmf.16

It seems to work properly and T267668 has not been reopened. If all goes well over night, I will deploy it on the rest of the wiki during the European time slot at 13:00 UTC (14:00 CET). Checking with SRE before pushing it obviously!