Moving to needs design, discussion with engineers needed.
FYI @Sneha you can safely ignore this ticket.

Almost no search queries on Commons contain custommatch:depicts_or_linked_from:

python
def collect_searches(spark):
    initial_query = """SELECT http, params
    FROM event.mediawiki_cirrussearch_request
    WHERE database='commonswiki' AND params IS NOT NULL
    """
    ddf = spark.sql(initial_query)
    filtered = (
        ddf
        .where(
            ddf.params.title.contains('Special:Search') | ddf.params.title.contains('Special:MediaSearch')
        )
        .where(
            ddf.http.request_headers.referer.contains('index.php')
        )
    )

We're focusing here on the following weighted tags that go to the Commons search index:

• image.linked.from.wikidata.p18/QID|SCORE
• image.linked.from.wikidata.p373/QID|SCORE
• image.linked.from.wikipedia.lead_image/QID|SCORE

where QID is a Wikidata item and SCORE is computed in commonswiki_file.py.

Looks like one AC is missing, moving back to ready for dev.

@BTullis this is done from the Structured Content team's side, so I'm removing tags.

Review done: https://gitlab.wikimedia.org/repos/structured-data/upload-tracking/-/merge_requests/3
Moving back to ready for dev

In T370106#10411930, @matthiasmullie wrote:

Thanks for catching that, Elena. I've submitted a patch that should fix that.

@Etonkovidova , it's merged.

In T381286#10411129, @Sneha wrote:

we don't do it for other input fields such as Authors, AI, where you found it etc...

Correct, I can confirm that.

In T381286#10410339, @Sneha wrote:

I think we only do that for date field right now.

I think we're doing that for pretty much all text inputs in the describe step, so different behavior in the release right step seems odd to me.

But are you saying that it would not clear the error as well if the user fixes the error?

No, errors are correctly cleared.

Moving to code review, but looking at Commons weighted tags usage in the meanwhile.

@Sneha , @matthiasmullie: both in own work and 3rd party, the custom license text inputs don't display errors as the user types, only when they hit the next button.
I think this should go to a different ticket, though.

@Htriedman I'll let you update https://gitlab.wikimedia.org/repos/security/differential-privacy/-/blob/main/.gitlab-ci.yml, CC @BTullis .

Filter out Commons while we figure out the importance of its weighted tags.

For 2024-11-25 snapshot we didn't have wmf.mediawiki_wikitext_current/snapshot=2024-11, so SLIS skipped. The SLIS sensor correctly failed, and the ALIS DAG completed, effectively shipping ALIS with no SLIS:

isu = spark.read.table('analytics_platform_eng.image_suggestions_suggestions').where('snapshot="2024-11-25"')
isu.where(isu.section_index.isNull()).count(), isu.where(isu.section_index.isNotNull()).count()

Published at https://meta.wikimedia.org/wiki/Machine_learning_models/Production/gogologo.
Closing.

• Indent all the sub-question boxes (1 in own-work flow and 2 in not-own-work flow) as shown in the UI

@Sneha not sure about this one: what exactly needs to be changed, if anything? The current production Commons already has indented boxes, and the patch doesn't seem to change that. Or maybe I'm not seeing obvious differences.

@Sneha @matthiasmullie :

• Update the warning copy under "not own work" > q1 option "I don’t know if it is free to share" as shown in the UI.

People will be happy to assist you at Wikimedia Commons's Village Pump. Thank you. is in patch set 1, but doesn't seem to be in the design: https://www.figma.com/design/PSsy485pa5YAiMsUrcoOui/Commons-upload-wizard?node-id=4362-21818&t=xbQmRcRVbbtM3fDv-4
I'll remove it.

All affected DAGs started today, closing.

In T372912#10336830, @dcausse wrote:

in the worst case scenario possibly but this means the score would have to change always in the same direction over time. I suspect (and ideally this would have to be tested) that the variations can go up or down so the gain I hope can be significant but I'm not familiar with your data so I can't judge.

Opened T380389: [SPIKE] Investigate search index delta variations.

alis.groupBy('snapshot').count().orderBy('snapshot').toPandas()

snapshot     count

0 2024-09-30 24284047
1 2024-10-07 24287195
2 2024-10-14 24290046
3 2024-10-21 24302041
4 2024-10-28 24329950
5 2024-11-04 24339009

Things we could do per wiki:

• [active wikis] compute a precision P based on user feedback, where P = accepted suggestions / ( accepted + rejected suggestions )

accepted = spark.sql("SELECT wiki, COUNT(is_accepted) AS accepted FROM event_sanitized.mediawiki_image_suggestions_feedback WHERE datacenter!='' AND year>=2022 AND month>0 AND day>0 AND hour<24 AND is_accepted=true GROUP BY wiki ORDER BY wiki").toPandas()
rejected = spark.sql("SELECT wiki, COUNT(is_rejected) AS rejected FROM event_sanitized.mediawiki_image_suggestions_feedback WHERE datacenter!='' AND year>=2022 AND month>0 AND day>0 AND hour<24 AND is_rejected=true GROUP BY wiki ORDER BY wiki").toPandas()
df = accepted.merge(rejected)
df['precision'] = df.accepted / (df.accepted + df.rejected)
df.sort_values('precision', ascending=False)

In T372912#10336760, @dcausse wrote:

to avoid sending too many updates we skip changes when the variation is within 20%, could this be something you could test and see how many updates we could save?

one issue I see here is: if we keep skipping small changes (when runs aren't skipped) then we'll always end up in huge updates, no?

In T372912#10336684, @dcausse wrote:

What's surprising is that "big" snapshots (which are assumed to be caused by missing upstream dependencies) can be ~40x bigger for commons but only 2x for enwiki.
@Cparle @mfossati is this something that can be understood and possibly fixed?

First thought off the top of my head is that Wikipedias get arrays of size 1 with exists|1 boolean tags, while Commons get arrays of size N with Wikidata item|score ones, which may be subject to a higher variation depending on Wikidata and how those scores are computed.