Risky Patch! 🚂🔥

• Change: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/960643
• Summary: Key format migration for CentralAuth central sessions and central tokens, step 1. Key migrations are hard to reason about. CentralAuth uses these keys to communicate cross-wiki so it's extra hard.
• Test plan: After group 0 and group 1 deploys, check that central login, edge login and central autologin still works, and check central tokens with the action and REST APIs.
• Places to monitor:
  • Logstash: mediawiki-errors
  • Grafana: authentication-metrics
• Revert plan: Revert the patch. If that doesn't fix things, we'll see. Central tokens are short-lived so fallout is minimal. Central sessions live for 24 hours; a botched migration might prevent logins, in the worst case might log people out. Not a big deal on group 0; not great on group 1 (loginwiki is group 1 so the group 0 and group 1 train will test different code paths; group 2 train should be no different from group 0 train). The patch is written defensively, and problems would only affect people who logged in / refreshed their session between the deploy and the revert, so the risk is low, especially if this is done not too long after the train.
• Affected wikis: group 0 train will affect group 0 wikis. Group 1 train will affect all wikis (but only for logins; group 1 wikis only for already existing sessions).
• IRC contact: tgr (@Tgr) (UTC-8), xSavitar (@xSavitar) (UTC+1)
• UBN Task Projects/tags: MediaWiki-Platform-Team
• Would you like to backport this change rather than ride the train?: No

T350627 is a deploy blocker. It seems to have been caused by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/939783
@jnuche please do not roll out the train until the patch has either been reverted or a suitable fix has been applied to the mobile site.

Change 971436 had a related patch set uploaded (by TrainBranchBot; author: trainbranchbot):

[mediawiki/core@wmf/1.42.0-wmf.4] Branch commit for wmf/1.42.0-wmf.4

https://gerrit.wikimedia.org/r/971436

Change 971436 merged by jenkins-bot:

[mediawiki/core@wmf/1.42.0-wmf.4] Branch commit for wmf/1.42.0-wmf.4

https://gerrit.wikimedia.org/r/971436

Change 972081 had a related patch set uploaded (by TrainBranchBot; author: MediaWiki PreSync):

[operations/mediawiki-config@master] testwikis wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972081

Change 972081 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972081

Mentioned in SAL (#wikimedia-operations) [2023-11-07T04:02:36Z] <mwpresync@deploy2002> Started scap: testwikis wikis to 1.42.0-wmf.4 refs T350080

Mentioned in SAL (#wikimedia-operations) [2023-11-07T04:53:40Z] <mwpresync@deploy2002> Finished scap: testwikis wikis to 1.42.0-wmf.4 refs T350080 (duration: 51m 04s)

Change 972328 had a related patch set uploaded (by TrainBranchBot; author: Jaime Nuche):

[operations/mediawiki-config@master] group0 wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972328

Change 972328 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972328

Mentioned in SAL (#wikimedia-operations) [2023-11-07T09:14:46Z] <jnuche@deploy2002> rebuilt and synchronized wikiversions files: group0 wikis to 1.42.0-wmf.4 refs T350080

Change 972697 had a related patch set uploaded (by TrainBranchBot; author: Jaime Nuche):

[operations/mediawiki-config@master] group1 wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972697

Change 972697 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972697

Change 972699 had a related patch set uploaded (by TrainBranchBot; author: Jaime Nuche):

[operations/mediawiki-config@master] group0 wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972699

Change 972699 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972699

Change 972747 had a related patch set uploaded (by TrainBranchBot; author: Jaime Nuche):

[operations/mediawiki-config@master] group1 wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972747

Change 972747 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972747

Mentioned in SAL (#wikimedia-operations) [2023-11-08T11:58:24Z] <jnuche@deploy2002> rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.4 refs T350080

Change 972816 had a related patch set uploaded (by TrainBranchBot; author: Jaime Nuche):

[operations/mediawiki-config@master] group0 wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972816

Change 972816 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972816

Mentioned in SAL (#wikimedia-operations) [2023-11-08T13:10:50Z] <jnuche@deploy2002> rebuilt and synchronized wikiversions files: group0 wikis to 1.42.0-wmf.4 refs T350080

Change 972861 had a related patch set uploaded (by TrainBranchBot; author: Jaime Nuche):

[operations/mediawiki-config@master] group1 wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972861

Change 972861 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/972861

@jnuche I was talked into making T350836: OAuth login to wikitech fails when running MediaWiki 1.42.0-wmf.4 a train blocker by @thcipriani and @Krinkle. The next scap of extensions/OAuth/src/SessionProvider.php on the 1.42.0-wmf.4 branch will break wikitech logins again. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/OAuth/+/972913 is the hack I'm currently proposing to fix that, but it needs smarter folks like @Tgr and @tstarling to decide if that hack is correct.

The new train blocker gives us two options:

1. Hold back labswiki (pin to wmf.3) and continue on to group2
2. Rollback to group0 and hope the OAuth problem is resolved

For option 1: I'm unaware of any way to hold back a single wiki using our fancy scap train command (might be wrong there). The way then to do this is the way we've done it in the past:

• scap update-wikiversions group2 1.42.0-wmf.4 (edits wikiverisons.json locally and that's it)
• manually edit /srv/mediawiki-staging/wikiversions.json so that labswiki is still on 1.42.0-wmf.3
• scap sync-wikiverisons
• push up the new wikiversions.json to gerrit so it persists.

On English Wiktionary, one of our gadgets suddenly broke. This brokenness was first reported at 18:52 UTC and I was able to resolve it by removing a reference to $.isReady from the gadget. See discussion.

Was jQuery upgraded in this MW version or something?

In T350080#9318075, @TTO wrote:

On English Wiktionary, one of our gadgets suddenly broke. This brokenness was first reported at 18:52 UTC and I was able to resolve it by removing a reference to $.isReady from the gadget. See discussion.

Was jQuery upgraded in this MW version or something?

Sorry to hear about the breakage, glad you were able to fix it. The full list of changes for this release is on: https://www.mediawiki.org/wiki/MediaWiki_1.42/wmf.4/Changelog

Change 973118 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group2 wikis to 1.42.0-wmf.4 (labswiki staying at 1.42.0-wmf.3 due to T350836)

https://gerrit.wikimedia.org/r/973118

Change 973118 merged by jenkins-bot:

[operations/mediawiki-config@master] group2 wikis to 1.42.0-wmf.4 (labswiki staying at 1.42.0-wmf.3 due to T350836)

https://gerrit.wikimedia.org/r/973118

SAL ended up here: https://phabricator.wikimedia.org/T350836#9318538

Mentioned in SAL (#wikimedia-operations) [2023-11-09T09:41:38Z] <jnuche@deploy2002> rebuilt and synchronized wikiversions files: Deploy 1.42.0-wmf.4 to group2 (labswiki staying at 1.42.0-wmf.3 due to T350836)

Change 973247 had a related patch set uploaded (by BryanDavis; author: Tim Starling):

[mediawiki/extensions/OAuth@wmf/1.42.0-wmf.4] Fix BlockDisablesLogin recursion

https://gerrit.wikimedia.org/r/973247

Late Thursday notes for @jnuche

There's a fix for wikitech, but it came in a little late in the day Thursday to sling it out.

Steps per @bd808 in IRC:

1. Deploy patch to wmf.4
2. Roll forward labswiki to wmf.4
3. If you can login to https://horizon.wikimedia.org with 2fa — SUCCESS!
4. Otherwise you can revert the change to wikiversions.json and run scap sync-wikiverisons again to go back to the state we're in today.

Check in the #wikimedia-cloud-admin channel if you need back up.

Also, I note, that Friday is a US holiday, so many folks won't be around later in your day. If you'd like to defer until Monday, that's a valid outcome.

Thanks for helming a tricky train!

@thcipriani since it looks like there's no hurry to get the fix out, I'd rather wait until Monday so @bd808 and anyone else needed are around just in case more fixes are required.

Thanks for your help and enjoy your long weekend!

Change 973247 merged by jenkins-bot:

[mediawiki/extensions/OAuth@wmf/1.42.0-wmf.4] Fix BlockDisablesLogin recursion

https://gerrit.wikimedia.org/r/973247

Mentioned in SAL (#wikimedia-operations) [2023-11-13T09:06:42Z] <jnuche@deploy2002> Started scap: Backport for [[gerrit:973247|Fix BlockDisablesLogin recursion (T350836 T350080)]]

Mentioned in SAL (#wikimedia-operations) [2023-11-13T09:08:34Z] <jnuche@deploy2002> bd808 and jnuche: Backport for [[gerrit:973247|Fix BlockDisablesLogin recursion (T350836 T350080)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-11-13T09:14:32Z] <jnuche@deploy2002> Finished scap: Backport for [[gerrit:973247|Fix BlockDisablesLogin recursion (T350836 T350080)]] (duration: 07m 49s)

Mentioned in SAL (#wikimedia-operations) [2023-11-13T09:31:19Z] <jnuche@deploy2002> rebuilt and synchronized wikiversions files: labswiki to 1.42.0-wmf.4 (T350836 T350080)

Confirmed it's possible to log into https://horizon.wikimedia.org after deploying https://gerrit.wikimedia.org/r/973247 and setting labswiki version to 1.42.0-wmf.4.

Change 973724 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] labswiki to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/973724

Change 973724 merged by jenkins-bot:

[operations/mediawiki-config@master] labswiki to 1.42.0-wmf.4

https://gerrit.wikimedia.org/r/973724