Wikipedia:Bots/Requests for approval/MusikBot 7

The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA. The result of the discussion was

Approved.

MusikBot 7

Operator: MusikAnimal (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)

Time filed: 02:08, Monday, November 16, 2015 (UTC)

Automatic, Supervised, or Manual: Automatic

Programming language(s): Ruby

Source code available: GitHub

Function overview: Carries out most of the weekly tasks involved with the Wikipedia:Today's articles for improvement project

Links to relevant discussions (where appropriate): Special:PermaLink/690468498#Bot automation

Edit period(s): Weekly

Estimated number of pages affected: 7

Exclusion compliant (Yes/No): Yes

Already has a bot flag (Yes/No): Yes

Function details: There are a lot of elements to this task, but the implementation is fairly straightforward. I will run through the process, start to finish:

Add a new "scheduled article" to Wikipedia talk:Today's articles for improvement. This involves only adding {{subst:TAFI scheduled selection}} to the end of the 2nd section. A Lua module in the template generates what will be the new TAFI in the coming weeks.
Check Wikipedia talk:Today's articles for improvement to see what was chosen as the new TAFI, and remove it from the list of articles at Wikipedia:Articles for improvement
Set up the schedule pages for the new TAFI, as described at Wikipedia:Today's articles for improvement/Schedule#Preparing a scheduled week. The bot creates the base page ([[Wikipedia:Today's articles for improvement/new_date_year/new_date_week]]) where new_date is the current date + 3 weeks. The subpage /1 that contains the name is also created. The bot does not attempt to choose an image and caption for the article (yet!)
Check [[Wikipedia:Today's articles for improvement/current_year/current_week/1]] to find out what the current TAFI is, and add {{TAFI}} to the top of that article. Theo's Little Bot currently does this, but apparently not all the time. The operator is inactive, and since MusikBot will be doing the whole process we figure it should take on this little task as well.
Notifying relevant WikiProjects that the new TAFI is within their scope. The bot does this by:
1. Checking the talk page of the article, using the API to get it's raw parsed HTML
2. Scan the markup for hyperlinks linking to Wikipedia:WikiProject_, using regex to grab the WikiProject_whatever part. I've found this to be the most reliable way to identify which WikiProjects we're working with. Categories and the names of the templates vary too much and are unreliable, it appears
3. Add {{subst:TAFI project notice}} to those WikiProject talk pages
Send a mass message to everyone on the notification list with the message body {{subst:TAFI weekly selections notice}}
Check [[Wikipedia:Today's articles for improvement/last_week_year/last_week/1]] for last week's TAFI. Then, remove {{TAFI}} from that article
Add {{Former TAFI}} to the article's talk page. This involves a few things:
1. Check the revision of the talk page a week prior and get the article assessment (stub, start, etc.)
2. Get the revision ID of the article a week prior
3. Get the current revision ID of the article and current assessment
4. Add {{Former TAFI}} with the appropriate values. If the class assessment has gone unchanged these parameters are not passed to the template
Update the accomplishments page using the {{Wikipedia:Today's articles for improvement/Accomplishments/row}} template.

This is all done in under 300 lines of code (a quarter of which is just handling API errors =P). I've tested as much as I can on testwiki, but obviously the real testing will have to be done in production. For the trial I will run each subtask individually, fully-monitored.

Note also there's more TAFI weekly tasks to come... but I figured this is a sizable "start", to say the least. — MusikAnimal ^talk 02:08, 16 November 2015 (UTC)[reply]

Discussion

This is very much a housekeeping task, so there's not much room for problems as long as you follow what's being requested. I don't like giving long trials, but I figure one is necessary here since this is a weekly task. Theo's Little Bot has options to disable individual tasks, so you can use those to turn off the particular one(s) that are being replaced. Approved for trial (4 weeks). Please provide a link to the relevant contributions and/or diffs when the trial is complete. — Earwig ^talk 07:07, 16 November 2015 (UTC)[reply]
@The Earwig: Note the new #5 and #6 subtasks I just added. They're coded and ready to try out. I figure if we're doing a 4-week trial I should squeeze in all of the weekly tasks into a single BRFA. I'm also going to add a on-wiki config page where you could turn off any given subtask. We just missed the mark for this week but the bot will be ready to go (supervised) for next Sunday at 00:00 GMT. Thanks! — MusikAnimal ^talk 16:07, 16 November 2015 (UTC)[reply]
Noting I added another task, #9 above. This leaves one more to tackle, which I hope to have done by round two of the trial next week. — MusikAnimal ^talk 05:26, 23 November 2015 (UTC)[reply]
Wait a sec, there's no such thing as Week 53. Surely the bot should've scheduled Person of the Year to Week 1 of 2016? Qwertyxp2000 (talk | contribs) 21:32, 8 December 2015 (UTC)[reply]
Well... — Earwig ^talk 21:37, 8 December 2015 (UTC)[reply]
Oh, either 52 or 53. Sorry I didn't realise this. I was just too systematic. Qwertyxp2000 (talk | contribs) 22:06, 8 December 2015 (UTC)[reply]
The Epoch converter is a pretty neat document, though with Ruby you can just do DateTime.now.cweek (calendar week). I can never remember those strftime directives. Also how cool is it that Person of the Year will coincidentally be the TAFI over new years? =P — MusikAnimal ^talk 23:58, 8 December 2015 (UTC)[reply]

Trial results

Trial complete. Not going to lie, getting this thing off the ground was rough, for an array of reasons. Here's a rundown of what happened:

Week 1 (23 November)

Did not go well. A little embarrassing, but here we go:

First and foremost, both MediaWiki and the labs server consider Monday to be the first day of the week. I did some tests and could have sworn it was Sunday. The task was quickly aborted once I realized this, and attempted again a day later.
{{{week}}} and others are not magic words, they are parameters passed to {{Wikipedia:Today's articles for improvement/Schedule/Preload}}. Stupid mistake. I thought maybe on testwiki these magic words just didn't exist. Lesson learned, and also all magic words are wrapped in two curly braces, not three, so there should be no confusion.
Some issues with adding {{Former TAFI}} to article pages, after a few tries this was fixed.
Some issues with adding the accomplishments, which were fixed.

Week 2 (30 November)

After doing some refactoring of the code, the start date passed to {{TAFI scheduled selection}} ended up being wrong. Fixed.
First time notifying WikiProjects was attempted. The {{TAFI project notice}} auto-supplies a heading, when the API wants it's own heading. This was not clear to me, and we ended up with duplicate headings [1]. I repaired those pages manually.

Week 3 (7 December)

For the TAFIDaily task, I changed the location of the list of nominations to be on a separate page. The {{TAFI scheduled selection}} Lua module was still looking in the wrong place, however, producing this error.
A major issue with my bot in general was discovered while testing the TAFIDaily task. That is, if I fetch the page properties in a certain way using the mediawiki-gateway gem, it escapes everything, when doing it another way does not. When I save to the page, the bot would always unescape everything. This is not a problem unless there are HTML entities somewhere on the page. The cross-task bug fix left some parts of the code accidentally unfixed, leading to this when it should be this. Ultimately I got everything squared away.
The mass messages for week 2 and 3 did not contain a timestamp, meaning bots will not archive them.

Week 4 (14 December)

The cronjob didn't start due to a typo.
I did not update the mediawiki-gateway gem to point to the newest version on GitHub, which had the code for mass messaging. Prior to this it was using the copy of the gem on my local machine. Problem fixed, and I re-ran that mass message task without any issues, and also this time a timestamp was included in the message.
The added scheduled selection at WT:TAFI had links to 2015 instead of 2016. For once this was not my fault, but a bug in the template, which I've fixed.

Takeaways

Special:Import exists, and can optionally have it import any templates used on the _target as well. This means I don't need to do the tedious work of copying/pasting each page from enwiki over to testwiki when testing the bot. Much of the issues outlined above would not have happened if all the relevant pages and templates were in place.
Do development on testwiki, but all-out testing should happen against enwiki, adding a debugger where the bot would normally save the page. This way you know exactly how the bot will perform in production, but we can catch and fix any bugs in the process without causing disruption. Had I thought of this sooner I think there would have been near to no bad edits during this trial, and the same is true with all of my other bot tasks.

All in all, I'm not entirely pleased with the trial, but am confident things are finally right. The only worry is as of yet, the TAFIWeekly task has not been ran by a cronjob, rather manually invoked. Next week (21 December) I hope to do this final test. The week following (28 December) I will be off-wiki and unable to monitor the bot for errors. If next week goes good, I think we'll be alright. I will be sure to inform the TAFI team that I will be away during the 28 December run.

So at the very least, I'm asking for one more run next week to test via a cronjob. Many thanks for your time! — MusikAnimal ^talk 04:45, 14 December 2015 (UTC)[reply]

Whoa. Well, I'm gonna admit difficulty reviewing this trial as I'm not involved with the project. @Northamerica1000, Evad37, SMcCandlish, Coin945, and Qwertyxp2000: could you folks chime in? Is it working as expected? Any potential issues? — Earwig ^talk 21:57, 15 December 2015 (UTC)[reply]

Well, there was a problem when MusikBot put in Week 1 in 2015 after Week 53 at one point. Well, the potential problems with reorganising the Level 2 titles is that I cannot find the correct sections well enough. I usually scroll down the WP:TAFINOM page to add another nomination to one of the alphabetised Level 2 titles. Qwertyxp2000 (talk | contribs) 22:09, 15 December 2015 (UTC)[reply]

@Qwertyxp2000: The rotating of the level 2 headings you speak of are part of the TAFIDaily task, and this was per request. Apparently the rotation was done in the past by Theo's Little Bot, and it stopped for whatever reason. If the TAFI team doesn't want that to happen that's no problem, it can be turned off in the config.

The other issue you speak of happened in week 2, I think, where it was bot operator stupidness. There was a lot of refactoring happening and this issue slipped through, and was manually fixed. Again there's so many moving parts to this task, so obviously it's hard for anyone to keep track of, which is why we want it automated! :) — MusikAnimal ^talk 22:35, 15 December 2015 (UTC)[reply]

Did want to mention however there is still the navigation at the top and the bottom of the nomination board that is still alphabetized. You can use this to quickly find the section you're looking for — MusikAnimal ^talk 22:36, 15 December 2015 (UTC)[reply]

Week 4 (14 December) step-by-step

I thought maybe if list out the most recent run, step by step, it might help you all evaluate the bot:

Add new scheduled selection to WT:TAFI [2]. Notice the picture and caption links are not red-linked as they should be (since those subpages aren't created by the bot). This was because {{TAFI scheduled selection}} was hardcoded to 2015 and not the current year (not the bot's fault). I've fixed this
Create subpages for the new scheduled selection (except picture and caption) [3] [4]
Remove the new scheduled selection from the list [5]
Add new scheduled selection to the archives [6] (apparently this made an extraneous entry in 2015, which is an easy fix) [7]
Tag current AFI [8]
Detag last week's AFI [9]
Add {{Former TAFI}} to last week's AFI talk page, including the class assessment, if it changed at all (here it did not) [10]
Notify relevant WikiProjects [11]
Add accomplishments [12]
Mass message project members [13]

So here the bot did everything it was supposed to (correct me if I'm wrong!), with exception of the bug in #4 which I've only now become aware of. I can simulate this on testwiki and ensure it is fixed (it just needs to go off the first embedded mb-bot comment and not all of them). Finally, as noted above, this did not run via a cronjob on WMF Labs. That's the last remaining piece as far as I can tell. So clearly an extension to the trial is needed, but a mere two weeks at most. We can skip the last week of 2015 and allow it to be manually done, if you want, since I won't be around.

Thanks to everyone for both their help and patience in working with me on this! — MusikAnimal ^talk 01:04, 16 December 2015 (UTC)[reply]

Umm... In WP:TAFIACCOMP, the dates consistently put either, for example, 9 Aug 2014 or 9 Aug 2015. Qwertyxp2000 (talk | contribs) 06:28, 16 December 2015 (UTC)[reply]

I see what you mean, it should use the abbreviated month name and not the full month name. No problem — MusikAnimal ^talk 15:59, 16 December 2015 (UTC)[reply]

{{BAGAssistanceNeeded}} Hoping to allow a one/two week extension to prove this bot does the job (see week 4 above, and the minor fixes that went with it). If an extension is not approved by Sunday 00:00 GMT, we will need to do all of the above manually. Please let me know so I can notify the TAFI team. Thanks! — MusikAnimal ^talk 19:20, 19 December 2015 (UTC)[reply]

Approved. Those edits all look fine to me, and there are no objections or apparent unresolved issues. If further problems come up, they can be easily dealt with. — Earwig ^talk 10:20, 20 December 2015 (UTC)[reply]

The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA.