mwapi calls rarely return results
Closed, ResolvedPublic8 Estimated Story PointsBUG REPORT
Actions

Assigned To

Authored By

	Svizel_pritula
	Sep 27 2020, 5:20 PM

Description

Steps to Reproduce:

Head to https://query.wikidata.org/ and open the "Filter labels using EntitySearch from mwapi service to provide Full Text Search" example.
Use the "Code" button to generate an URL for this query. Add &format=json at the end to allow viewing the result in a web browser. You should arrive at this URL.
Spam CTRL + F5 to force refresh the page a bunch of times (usually at least 10) and take note of the results.

Actual Results:

Most of the time, the results will be empty. Sometimes, however, the correct results are returned.

Expected Results:

The results are always correctly populated.

Details

	Subject	Repo	Branch	Lines +/-
	Better resource handling for mwapi calls	wikidata/query/rdf	master	+12 -2
	Add more debug logging in MWApiServiceCall	wikidata/query/rdf	master	+4 -0

Customize query in gerrit

Event Timeline

Svizel_pritula created this task.Sep 27 2020, 5:20 PM

Restricted Application added a project: Wikidata. · View Herald TranscriptSep 27 2020, 5:20 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Svizel_pritula updated the task description. (Show Details)Sep 27 2020, 5:20 PM

CBogen moved this task from Incoming to Current work on the Wikidata-Query-Service board.Sep 28 2020, 3:24 PM

CBogen added a project: Discovery-Search (Current work).

CBogen set the point value for this task to 8.Sep 28 2020, 5:26 PM

CBogen moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.

Daniel_Mietchen subscribed.Sep 28 2020, 10:04 PM

• dcausse claimed this task.Sep 29 2020, 4:19 PM

• dcausse moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.

Dipsacus_fullonum subscribed.Sep 30 2020, 11:40 AM

abian awarded a token.Sep 30 2020, 8:05 PM

abian subscribed.

Kdutia subscribed.Oct 1 2020, 8:47 AM

Change 631492 had a related patch set uploaded (by DCausse; owner: DCausse):
[wikidata/query/rdf@master] Add more debug logging in MWApiServiceCall

https://gerrit.wikimedia.org/r/631492

gerritbot added a project: Patch-For-Review.Oct 1 2020, 3:36 PM

Change 631492 merged by jenkins-bot:
[wikidata/query/rdf@master] Add more debug logging in MWApiServiceCall

https://gerrit.wikimedia.org/r/631492

Maintenance_bot removed a project: Patch-For-Review.Oct 1 2020, 5:10 PM

Zache subscribed.Oct 1 2020, 5:56 PM

The root cause of the problem is yet unclear.
Added some more debug logs to continue investigating.
What I know so far is that only codfw was affected and restarting blazegraph on an affected node fixed the issue. A state is probably leaked but it's unclear where yet, could be in blazegraph itself or in the jetty http client (the additional logging should hopefully help to discard one option or another).

Mentioned in SAL (#wikimedia-operations) [2020-10-21T14:34:30Z] <dcausse> restarting blazegraph on codfw servers (T263952)

resuming investigation, additional logs seem to suggest that the jetty http client (or the way we use it) is to blame.

Change 636032 had a related patch set uploaded (by DCausse; owner: DCausse):
[wikidata/query/rdf@master] Better resource handling for mwapi calls

https://gerrit.wikimedia.org/r/636032

gerritbot added a project: Patch-For-Review.Oct 23 2020, 2:11 PM

Change 636032 merged by jenkins-bot:
[wikidata/query/rdf@master] Better resource handling for mwapi calls

https://gerrit.wikimedia.org/r/636032

• dcausse moved this task from In Progress to To Be Deployed on the Discovery-Search (Current work) board.Oct 26 2020, 6:07 PM

• dcausse moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Nov 2 2020, 6:04 PM

This should be resolved, but hard to reproduce, so hard to test. Feel free to reopen if the error occurs again.

mwapi calls rarely return resultsClosed, ResolvedPublic8 Estimated Story PointsBUG REPORTActions

Description

Details

Event Timeline

mwapi calls rarely return results
Closed, ResolvedPublic8 Estimated Story PointsBUG REPORT
Actions