Page MenuHomePhabricator
Create Task
Maniphest T205252

Reduce text returned by EntityContent::getTextForFilters
Closed, ResolvedPublic
Actions

Assigned To
Addshore
Authored By
Addshore
Sep 24 2018, 8:59 AM
Tags
Referenced Files
None
Subscribers
Addshore
Aklapper
DannyS712
gerritbot
Krinkle
Lea_Lacroix_WMDE
Tokens
"Like" token, awarded by Pintoch.

Description

T204109 profiles a 30 seconds API edit request and determines that roughly 10 seconds of that time is spent in the AbuseFilter edit filter hook.
T204109#4602542 looks at how many more elements can possible be ignored when generating the text that is passed to the AbuseFilter.
Ignoring more elements not only speeds up the execution time of the collection of strings, but will also have knock on effects of the time taken to process all of the strings.

We should investigate which strings are actually used by the community in AbuseFilter rules and then remove the elements that are not needed.

Details

SubjectRepoBranchLines +/-
Add LexemeContent::getIgnoreKeysForFiltersmediawiki/extensions/WikibaseLexememaster+12 -0
LexemeContent::getTextForFilters testsmediawiki/extensions/WikibaseLexememaster+132 -0
Add Item/PropertyContent::getIgnoreKeysForFiltersmediawiki/extensions/Wikibasemaster+27 -9
EntityContent::getTextForFilters testsmediawiki/extensions/Wikibasemaster+253 -0
Add MediaInfoContent::getIgnoreKeysForFiltersmediawiki/extensions/WikibaseMediaInfomaster+10 -0
MediaInfoContent::getTextForFilters testsmediawiki/extensions/WikibaseMediaInfomaster+105 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Addshore triaged this task as Medium priority.Sep 24 2018, 8:59 AM
Addshore created this task.
Restricted Application added a project: User-Addshore. · View Herald TranscriptSep 24 2018, 8:59 AM
Addshore changed the task status from Open to Stalled.Sep 24 2018, 9:00 AM

Stalled as it requires community investigation to be done first.

Addshore removed Addshore as the assignee of this task.Sep 24 2018, 9:00 AM
Addshore removed a project: User-Addshore.
Addshore mentioned this in T205254: Investigate usage of "text" in AbuseFilter rules on wikidata.org.Sep 24 2018, 9:06 AM
Addshore moved this task from Inbox to Investigate & Discuss on the [DEPRECATED] wdwb-tech board.
Addshore moved this task from Investigate & Discuss to Triaged High (100+) on the [DEPRECATED] wdwb-tech board.Oct 8 2018, 12:45 PM
Addshore moved this task from incoming to needs discussion or investigation on the Wikidata board.Oct 8 2018, 1:05 PM
Addshore added a project: Technical-Debt.Oct 31 2018, 4:53 PM
gerritbot added a comment.Nov 16 2018, 9:30 AM

Change 474135 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/WikibaseMediaInfo@master] Add MediaInfoContent::getIgnoreKeysForFilters

https://gerrit.wikimedia.org/r/474135

gerritbot added a project: Patch-For-Review.Nov 16 2018, 9:30 AM

Change 474136 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/WikibaseLexeme@master] Add LexemeContent::getIgnoreKeysForFilters

https://gerrit.wikimedia.org/r/474136

gerritbot added a comment.Nov 16 2018, 9:31 AM

Change 474137 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Add Item/PropertyContent::getIgnoreKeysForFilters

https://gerrit.wikimedia.org/r/474137

Addshore mentioned this in rEWBI161af0290bd9: Add MediaInfoContent::getIgnoreKeysForFilters.Nov 16 2018, 9:32 AM
Addshore mentioned this in rEWLE6542c6c69215: Add LexemeContent::getIgnoreKeysForFilters.
Addshore added a comment.Nov 16 2018, 9:37 AM

The 3 patches above move the definition of keys to ignore out of EntityContent and into each Content object itself so that they can be defined per entity type.

We still need T205254 to be done before we can remove any keys from the text we pass to abuse filters.

gerritbot added a comment.Nov 16 2018, 10:50 AM

Change 474147 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] EntityContent::getTextForFilters tests

https://gerrit.wikimedia.org/r/474147

gerritbot added a comment.Nov 16 2018, 10:57 AM

Change 474150 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/WikibaseMediaInfo@master] MediaInfoContent::getTextForFilters tests

https://gerrit.wikimedia.org/r/474150

Addshore mentioned this in rEWBIcb74f66bd55c: Add MediaInfoContent::getIgnoreKeysForFilters.Nov 16 2018, 10:58 AM
Addshore mentioned this in rEWBI2b8584730715: MediaInfoContent::getTextForFilters tests.
gerritbot added a comment.Nov 16 2018, 11:20 AM

Change 474155 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/WikibaseLexeme@master] LexemeContent::getTextForFilters tests

https://gerrit.wikimedia.org/r/474155

Addshore mentioned this in rEWLE202257fd00e6: LexemeContent::getTextForFilters tests.Nov 16 2018, 11:21 AM
Addshore mentioned this in rEWLEcf73f6303753: Add LexemeContent::getIgnoreKeysForFilters.
gerritbot added a comment.Nov 16 2018, 12:51 PM

Change 474150 merged by jenkins-bot:
[mediawiki/extensions/WikibaseMediaInfo@master] MediaInfoContent::getTextForFilters tests

https://gerrit.wikimedia.org/r/474150

gerritbot added a comment.Nov 16 2018, 12:51 PM

Change 474135 merged by jenkins-bot:
[mediawiki/extensions/WikibaseMediaInfo@master] Add MediaInfoContent::getIgnoreKeysForFilters

https://gerrit.wikimedia.org/r/474135

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.6; 2018-11-27).Nov 16 2018, 1:00 PM
gerritbot added a comment.Dec 3 2018, 2:26 PM

Change 474147 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] EntityContent::getTextForFilters tests

https://gerrit.wikimedia.org/r/474147

Lucas_Werkmeister_WMDE mentioned this in T204109: Investigate some Wikidata write api queries taking over 30 seconds. (In EntityContent::collectValues?).Dec 3 2018, 2:38 PM
ReleaseTaggerBot edited projects, added MW-1.33-notes (1.33.0-wmf.8; 2018-12-11); removed MW-1.33-notes (1.33.0-wmf.6; 2018-11-27).Dec 3 2018, 3:00 PM
gerritbot added a comment.Dec 4 2018, 10:41 AM

Change 474155 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] LexemeContent::getTextForFilters tests

https://gerrit.wikimedia.org/r/474155

gerritbot added a comment.Dec 4 2018, 10:41 AM

Change 474136 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] Add LexemeContent::getIgnoreKeysForFilters

https://gerrit.wikimedia.org/r/474136

gerritbot added a comment.Dec 4 2018, 10:41 AM

Change 474137 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add Item/PropertyContent::getIgnoreKeysForFilters

https://gerrit.wikimedia.org/r/474137

Addshore added a comment.Mar 28 2019, 10:53 AM

We got to a list of some things that could be removed in an initial iteration @ T205254#5047549

But....

Splitting out guid with the current code poses problems, as the key for the field is "id", but simply removing this at all levels is too heavy handed, as it would result in the ID for statement values being removed too etc.
If we want to decrease the amount of text sent to abusefilter we might have to slightly refactor the way the text is collected.
It might make sense to have a seperate set of serializers / codecs for representing an entity for abusefilter, then these could just be used..

Addshore changed the task status from Stalled to Open.Jun 20 2019, 9:53 PM
Addshore closed subtask T205254: Investigate usage of "text" in AbuseFilter rules on wikidata.org as Resolved.

Going to create some sub tickets.

Addshore mentioned this in T226216: Statement GUIDs should not appear in AbuseFilter text for Wikibase.Jun 20 2019, 10:06 PM
Addshore added a subtask: T226216: Statement GUIDs should not appear in AbuseFilter text for Wikibase.Jun 20 2019, 10:09 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 20 2019, 10:10 PM
Addshore added a subtask: T226218: snak hashes should not appear in AbuseFilter text for Wikibase.Jun 20 2019, 10:12 PM
Addshore moved this task from needs discussion or investigation to in progress on the Wikidata board.Jun 21 2019, 11:31 PM
Lea_Lacroix_WMDE subscribed.Jul 25 2019, 2:47 PM

Announced on the admin noticeboard, pinging the main AbuseFilter maintainers. Suggested them to make comments here, if nothing blocking we can merge on August 6th.

Pintoch awarded a token.Jul 25 2019, 2:50 PM
DannyS712 subscribed.Jul 26 2019, 5:41 AM
• alaa_wmde closed subtask T226216: Statement GUIDs should not appear in AbuseFilter text for Wikibase as Resolved.Oct 1 2019, 2:51 PM
Addshore closed subtask T226218: snak hashes should not appear in AbuseFilter text for Wikibase as Resolved.Dec 5 2019, 9:35 AM
Addshore closed this task as Resolved.Dec 5 2019, 9:39 AM
Addshore claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits
  NODES
admin 1
COMMUNITY 2
Note 5
Project 7