Change 666860 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] GrowthExperiments: link recommendation service URL for beta

https://gerrit.wikimedia.org/r/666860

Change 666860 merged by jenkins-bot:
[operations/mediawiki-config@master] GrowthExperiments: link recommendation service URL for beta

https://gerrit.wikimedia.org/r/666860

Change 666969 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Add verbose parameter to refreshLinkRecommendations.php

https://gerrit.wikimedia.org/r/666969

Change 666969 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Add verbose parameter to refreshLinkRecommendations.php

https://gerrit.wikimedia.org/r/666969

Change 655865 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/puppet@production] Add GrowthExperiments maintenance script

https://gerrit.wikimedia.org/r/655865

There are two problems with the beta setup:

• there are no ORES topics on beta. We'd have to set a bunch manually.
• most (all?) beta wikis use $wgGENewcomerTasksRemoteApiUrl which means the refreshLinkRecommendations.php candidate search will return results which don't exist locally. Probably time to do away with that config setting.

Change 667010 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] [beta] GrowthExperiments: disable remote API use on some wikis

https://gerrit.wikimedia.org/r/667010

In T274198#6862670, @Tgr wrote:

• there are no ORES topics on beta. We'd have to set a bunch manually.

The results for goodfaith model were randomly added to betalabs db ores_classification table. Would it be something as simple as that?

In T274198#6862813, @Etonkovidova wrote:

In T274198#6862670, @Tgr wrote:

• there are no ORES topics on beta. We'd have to set a bunch manually.

The results for goodfaith model were randomly added to betalabs db ores_classification table. Would it be something as simple as that?

Could we have a bash script that:

• iterates over ORES topic listing
  • iterates over task types
    • gets ~10 page IDs
• Does a SpecialExport call (not sure if there is an API for that, though)
• Runs the XML through maintenance/importDump.php

That would also be useful for local environments.

Change 667010 merged by jenkins-bot:
[operations/mediawiki-config@master] [beta] GrowthExperiments: disable remote API use on some wikis

https://gerrit.wikimedia.org/r/667010

These are the options I can think of:

• set the ORES topics manually (randomly, or by checking for a production page with the same name)
• use a prod/dev flag to make refreshLinkRecommendations.php make a generic search instead of per-topic searches
• set up some mechanism in beta to add random ORES tags when a page is created

The third is very simple to do but would only work on new pages. I would not add another dev/prod behavior fork if we can avoid it (plus being able to use ORES topics in the filter UI on beta does have some value). So I think a maintenance script for the first is the best option, and that could also incorporate importing articles.

T275909: CirrusSearch search results disappear on beta wiki when quoting the hastemplate: parameter value is a problem for the beta setup.

Also, right now I get "An error has occurred while searching: We could not complete your search due to a temporary problem. Please try again later." for any kind of search, but maybe that's a temporary issue.

In T274198#6868598, @Tgr wrote:

Also, right now I get "An error has occurred while searching: We could not complete your search due to a temporary problem. Please try again later." for any kind of search, but maybe that's a temporary issue.

T275975: Search broken on beta cluster wikis

Mentioned in SAL (#wikimedia-operations) [2021-03-01T12:10:30Z] <awight@deploy1001> Synchronized wmf-config: Config: [[gerrit:666441|GrowthExperiments: set GELinkRecommendationsUseEventGate (T274198)]] (duration: 01m 05s)

Change 667579 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] [DNM] Script for adding topics to a test environment

https://gerrit.wikimedia.org/r/667579

In T274198#6868249, @Tgr wrote:

These are the options I can think of:

• set the ORES topics manually (randomly, or by checking for a production page with the same name)
• use a prod/dev flag to make refreshLinkRecommendations.php make a generic search instead of per-topic searches
• set up some mechanism in beta to add random ORES tags when a page is created

The third is very simple to do but would only work on new pages. I would not add another dev/prod behavior fork if we can avoid it (plus being able to use ORES topics in the filter UI on beta does have some value). So I think a maintenance script for the first is the best option, and that could also incorporate importing articles.

Agreed on the maintenance script for setting topics based on prod equivalents and importing articles. Shall we make a new task for that?

In T274198#6870279, @kostajh wrote:

Agreed on the maintenance script for setting topics based on prod equivalents and importing articles. Shall we make a new task for that?

T276163: Create maintenance script for importing ORES topics

Change 655865 merged by Jcrespo:
[operations/puppet@production] Add GrowthExperiments maintenance script

https://gerrit.wikimedia.org/r/655865

@Tgr https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/667941 adds support for using a personal access token so that the maintenance script on beta could call the production service instead of the toolforge instance. I don't know about how we could store a personal access token on beta cluster, though.

In the private puppet, as long as we don't expect it to be too private (anyone with shell access to beta cluster will be able to check it).

@Tgr I've updated the description of the task, could you please have a look to see that if seems correct?

Yes, other than that some wikis have articles already so the first step is not necessarily needed.

The API token is stored in the deployment-deploy01:/srv/mediawiki-staging/private repo, in commit 1487f42589f0. In case it needs to be pulled for whatever reason, the management URL is https://meta.wikimedia.org/wiki/Special:OAuthManageConsumers/30651f7dcd295af70d0364c1dd3ae460.

Change 668470 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] Add Link: use production link recommendation service

https://gerrit.wikimedia.org/r/668470

Change 668470 merged by jenkins-bot:
[operations/mediawiki-config@master] [beta] Add Link: use production link recommendation service

https://gerrit.wikimedia.org/r/668470

Change 669062 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Add request timeout parameter for the link recommendation service

https://gerrit.wikimedia.org/r/669062

Change 669099 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Add $wgGEDeveloperSetup

https://gerrit.wikimedia.org/r/669099

With the search index definition fixed, the next issue is that refreshLinkRecommendations.php does not actually update the index. UpdateWeightedTags.php, which uses the same internal call, does work; and judging from the EventBus log, refreshLinkRecommendations.php does schedule the update job; but the link recommendation tag doesn't appear.

Also, the link recommendation service is very unreliable: after three normal responses it gives a long stream of 503s. (It seems like some sort of throttling, but then why an 503 instead of a 429?) And often those normal responses are 504s - envoy being strict with timeouts, I assume.

In T274198#6889955, @Tgr wrote:

With the search index definition fixed, the next issue is that refreshLinkRecommendations.php does not actually update the index. UpdateWeightedTags.php, which uses the same internal call, does work; and judging from the EventBus log, refreshLinkRecommendations.php does schedule the update job; but the link recommendation tag doesn't appear.

Scratch that, it seems to work, it just takes much longer for the updates to take effect than I'd have expected.

Change 669459 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Add fixLinkRecommendationData.php maintenance script

https://gerrit.wikimedia.org/r/669459

Change 669469 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] refreshLinkRecommendations.php: commit DB writes sooner

https://gerrit.wikimedia.org/r/669469

Not done yet, but the changes will be needed on beta to finish.

Change 669062 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Add request timeout parameter for the link recommendation service

https://gerrit.wikimedia.org/r/669062

Change 669099 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Add $wgGEDeveloperSetup

https://gerrit.wikimedia.org/r/669099

Change 669469 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] refreshLinkRecommendations.php: commit DB writes sooner

https://gerrit.wikimedia.org/r/669469

Change 669757 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/vagrant@master] growthexperiments: Set $wgGEDeveloperSetup

https://gerrit.wikimedia.org/r/669757

Change 669757 merged by jenkins-bot:
[mediawiki/vagrant@master] growthexperiments: Set $wgGEDeveloperSetup

https://gerrit.wikimedia.org/r/669757

Change 669459 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Add fixLinkRecommendationData.php maintenance script

https://gerrit.wikimedia.org/r/669459

Change 669816 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] [beta] GrowthExperiments: set $wgGEDeveloperSetup = true on beta

https://gerrit.wikimedia.org/r/669816

Change 669816 merged by jenkins-bot:
[operations/mediawiki-config@master] [beta] GrowthExperiments: set $wgGEDeveloperSetup = true on beta

https://gerrit.wikimedia.org/r/669816

Change 669879 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/mediawiki-config@master] [beta] GrowthExperiments: increase timeout for requests to service

https://gerrit.wikimedia.org/r/669879

Change 669879 merged by jenkins-bot:
[operations/mediawiki-config@master] [beta] GrowthExperiments: increase timeout for requests to service

https://gerrit.wikimedia.org/r/669879

@Tgr I've proposed English, Czech, Vietnamese, Arabic, and Bengali for set up, does that sound OK to you? Those are the ones we want to deploy to in production first, so doing early QA in beta would be nice. I think of these five we could prioritize Czech and Arabic.

Agreed, we should go for enwiki + the deployment _target. en and cs are the two I've tried first (mostly because it is easier to read page titles, so it is easier to debug).

The current status is:

• the service is apparently being throttled, so the refresh job has a very large error rate
• some tasks do succeed, but even those aren't added to the search index (the DB part works as expected). The Cirrus search index update job seems to be fired as exected so not sure where it gets lost.
• importOresTopics doesn't work super well because it goes through the same pages again and again. Fixing that requires T276827. Or we could make the script iterate through all pages or read them from a file.
• cs and end have now one-one page manually set up (ie. added to the search index via UpdateWeightedTags.php), one of them works, the other doesn't. That's T276838.

Change 670226 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/GrowthExperiments@master] Allow resetting the task cache with a query flag

https://gerrit.wikimedia.org/r/670226

Change 670226 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Allow resetting the task cache with a query flag

https://gerrit.wikimedia.org/r/670226

Cronjobs don't seem to be working on beta: T277206: Beta cron jobs seem broken

The script does not write to the search index correctly: T277208: Add Link: refreshLinkRecommendations.php does not write to the search index on beta

@Tgr anything else to be done here?

In T274198#6934014, @kostajh wrote:

@Tgr anything else to be done here?

See if beta en/vi/ar/bn have enough articles or need an import, import ORES topics, run the refresh script. Maybe set it up as a personal cronjob until T277206 gets fixed (sounds like it might take a while).

Did a full run of the refresh script on beta (which is configured to have 25 links per topic to make it take less long, so that's 1600 tasks in total):

• 10 topics have been filled
• the remaining 54 topics are, in total, 985 tasks short
• there were 421 timeouts
• there were 1037 pages where the recommendation had less than 4 links
• there were 428 pages where there was a recommendation in the database but not in the index (that's somewhat expected since index updates are slow, but that that's a suspiciously high number)
• 16 were excluded because they contained a link recommendation edit.

These are all articles where the title was imported from cswiki, so they are somewhat indicative of production cswiki performance, except of course there will be much more candidates there, and probably no timeouts. (Also I'm not sure if the text was also imported in all cases, so some of the "less than 4 links" could be spurious.)

212 tasks were added successfully (the reason the numbers don't quite add up is that I did some only-partially-successful test runs earlier and didn't save the output there), that's about 10% of all attempts. Hopefully that's only due to beta not having that many articles.

Change 675901 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[operations/mediawiki-config@master] [beta-only] Use local GrowthExperiments task suggester on more wikis

https://gerrit.wikimedia.org/r/675901

Change 675901 merged by jenkins-bot:

[operations/mediawiki-config@master] [beta-only] Use local GrowthExperiments task suggester on more wikis

https://gerrit.wikimedia.org/r/675901

Change 676843 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/CirrusSearch@master] ForceSearchIndex.php: fix --namespace option for main namespace

https://gerrit.wikimedia.org/r/676843

Change 676844 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/GrowthExperiments@master] Make importOresTopics.php work for wikis which do not have ORES models

https://gerrit.wikimedia.org/r/676844

This is now done for all five specified wikis, via the following steps:

# import articles
curl -s "https://${WIKI:0:2}.wikipedia.org/w/api.php?action=query&format=json&export=1&exportnowrap=1&exportschema=0.11&generator=random&grnfilterredir=nonredirects&grnnamespace=0&grnlimit=1000" > dump-$WIKI.xml
mwscript importDump.php $WIKI dump-$WIKI.xml --no-local-users --username-prefix $WIKI --report 50
mwscript rebuildrecentchanges.php $WIKI
mwscript initSiteStats.php $WIKI --update
mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php $WIKI --namespace 0 --skipLinks --indexOnSkip
mwscript extensions/CirrusSearch/maintenance/ForceSearchIndex.php $WIKI --namespace 0 --skipParse

# import ORES topics
# using https://pypi.org/project/wildq/
cat dump-$WIKI.xml | wq --xml '.mediawiki.page[].title' > pages-$WIKI.txt
mwscript extensions/GrowthExperiments/maintenance/importOresTopics.php $WIKI --pageList pages-$WIKI.txt
# wait an hour or so; for whatever reason, setting weighted tags in the search index is quite slow on beta. Then check topics:
mwscript extensions/GrowthExperiments/maintenance/listTaskCounts.php $WIKI --topictype ores --tasktype link-recommendation --output json | jq '.topicCounts | add'

# generate tasks
mwscript extensions/GrowthExperiments/maintenance/refreshLinkRecommendations.php $WIKI
# wait an hour or so like above; check tasks:
mwscript extensions/GrowthExperiments/maintenance/listTaskCounts.php $WIKI --topictype ores --tasktype link-recommendation --output json | jq '.taskTypeCounts["link-recommendation"]'

(The import step was skipped on en and cs which already had ~1000 articles.)

For cswiki and enwiki that went reasonably well - on cswiki something like 40% of the pages were valid tasks; the enwiki task generation script is still running but it seems like it will also result in hundreds of tasks. For vi/bn/ar though the number of pages which are valid tasks is tiny though; for 97-99% of pages the service recommends less than 4 links.

Thank you @Tgr!

@Etonkovidova please let us know if you need anything else here.

Change 676843 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] ForceSearchIndex.php: fix --namespace option for main namespace

https://gerrit.wikimedia.org/r/676843

Change 676844 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Make importOresTopics.php work for wikis which do not have ORES models

https://gerrit.wikimedia.org/r/676844