Work related to the WMF fork and deployment of the Wiki Education Foundation's dashboard app.
Past sprints: https://phabricator.wikimedia.org/project/board/1786/ and https://phabricator.wikimedia.org/project/board/1750/
The app runs in Cloud VPS, in the globaleducation project.
There is also the Toolforge tool wikiedudashboard which is running the code from WikiEduDashboardTools.
Amazing, thank you!
Yes, I think it was related. It was hanging on fetching two Vega libraries that are hosted on toolforge (instead of a commercial VPN). I stopped and started the webservice for wikiedudashboard tool, and that fixed it.
May be related to T383238: [nfs] 2025-01-08 tools-nfs outage? (I don’t know to what extent those URLs would rely on NFS.)
@empty-codes great job, thank you!
An 'Admin Guide' containing an overview of the Dashboard infrastructure, including servers, tools, dependencies, and troubleshooting resources has been created and linked to the project's meta wiki page. Closing the task as completed.
I'm going to say that CampaignEvents has fulfilled this request.
I've restarted the webserver and it's back up.
We've deploy an update that skips the requirement to remove the campaigns on P&E Dashboard.
Yes, thank you so much @Ragesoss
I think this would make a great task to roll into the (hopefully) upcoming Outreachy project: https://phabricator.wikimedia.org/T374390
Thanks @Ragesoss.
Either is fine. The system has some database bottlenecks related to running updates on courses/events with large numbers of edits, so when a new high-activity event (or a surge of activity for an ongoing one) hits the system, it sometimes puts enough load on the database that the web service gets backed up and becomes unresponsive. It typically comes back online within a few hours at most (or sooner if I reboot the VM that runs the webserver), but it's down enough that I know it causes problems for things like sharing metrics with external partners.
I wonder if this issue should also be reported here: https://meta.wikimedia.org/w/index.php?title=Talk:Programs_%26_Events_Dashboard&action=edit§ion=new
I've filed an issue for this. The requirement to remove the campaigns before deleting a course is the desired behavior on Wiki Education Dashboard, but I think it does make sense to remove the requirement on Programs & Events Dashboard.
FWIW, I can confirm that the event in question has been successfully unsynced (using the unsync_event endpoint) a few hours ago.
I restarted the webserver.
I restarted the web server. I'm not sure what caused this particular outage, but we've got plans to improve the monitoring so that it's easier to pinpoint performance bugs.
The website is back up now. ¯\_(ツ)_/¯