Wikipedia talk:New pages patrol/Analysis and proposal

Thank you

Thank you to the folks who put this together. I quickly scanned the analysis, and while I don't agree with a few of its assumptions, it does shine a light on the need to find a balance between thoroughness and speed. I will probably have more comments or questions after I read it more carefully.- MrX 23:43, 31 May 2017 (UTC)

Graph question

Thanks so much for this thorough consideration. I look forward to thinking through and discussing together. If I may ask one informational point to begin (forgive me if this should be obvious). As to the key in the graph, saying blue indicates "users who are still new (not autoconfirmed) today". Does blue then indicate only from pages from editors who were still not autoconfirmed when you generated this graph on May 25, or all pages from editors not autoconfirmed at the time of their entry's creation? I.e. if four months ago, a not-yet-autoconfirmed editor made a page; it's still in the backlog; but that person subsequently made ten edits to WP, becoming autoconfirmed: is their backlogged entry marked green or blue in the graph? I ask because this would, of course, change how much the graph tells us about what percentage of the backlog would be affected by requiring autoconfirm at the time the editor creates the page.

Thanks again for the sustained attention to this important challenge. Innisfree987 (talk) 23:46, 31 May 2017 (UTC)

Yes, thanks for asking. We're working on a second version that looks at the time of creation; we should have it pretty soon, and we'll add it to the report. DannyH (WMF) (talk) 01:04, 1 June 2017 (UTC)

Great, thanks! Innisfree987 (talk) 01:19, 1 June 2017 (UTC)

@Innisfree987: To answer your question, the blue part of the graph indicates pages created by editors who were still not autoconfirmed when we generated the graph on May 25. Unfortunately, it's quite difficult to generate numbers based on the editor's status at the time of article creation, but we are still working on trying to get numbers for that. Kaldari (talk) 20:29, 1 June 2017 (UTC)

Yes, "today" in that legend refers to current times (more precisely, 0:38 UTC on May 25), so your first interpretation is correct. To explain why the graph uses that definition, and also to clear up some misunderstandings elsewhere:

This graph represents a snapshot of the existing review backlog as it appears to new page patrollers at Special:NewPagesFeed. In particular, it uses the same definition of "new editors". I.e. if you change settings there to only show unreviewed articles by "new editors", you should get a list that exactly corresponds to the blue part of the graph.

As to why this existing backlog display interface at Special:NewPagesFeed uses a definition of "new editors" based on their autoconfirmed status today rather than at the time of article creation: This is a deliberate choice in the software - it contains code written specifically for the purpose of updating an article creator's status in Special:NewPagesFeed once they reach autoconfirmed status after creating the article. I wasn't around in 2012 when this decision was made, but I can imagine various arguments for and against it. If people want to make a case that the other alternative (status at article creation) would actually be more useful for patrollers, I imagine that a Phabricator request to change it would receive fair consideration.

Back to the chart: As Danny says, we had already been working on a version based on the alternative definition, because that's more pertinent to the specific question of how much the backlog would be reduced if non-autoconfirmed users were prevented from creating articles. (I have by now written and tested a query for this. But based on some speed tests on a smaller amount of data, it may take several days to run fully, and could make our DBAs unhappy, so I'm still working on optimizing it further. In any case, we should have a result next week.)

However, we don't expect the result of this refined query to differ too much regarding the overall takeaway that new users - by either definition - are only responsible for a small part of the backlog. This is based, for example, on a manual check I did last week, examining 65 articles from the backlog (selected arbitrarily from those created on February 15 and March 15), of which only 3 were by new users who had subsequently become autoconfirmed. That's why we felt comfortable using the first version of the chart (the one consistent with Special:NewPagesFeed) in this report already.

PS: This is all based on public data, and the PAWS notebook used to create the graph is linked in the report, so anyone can reproduce and modify it - I'm happy to help. I think the graph's legend was fairly unambiguous already, but I also just updated the file description page with more detail and the link to the notebook. Regards, Tbayer (WMF) (talk) 01:45, 2 June 2017 (UTC)

Tbayer (WMF) and Kaldari, thank you both for the detailed information about the data available. All makes sense; in fact I'm pleasantly surprised it's possible, I wasn't sure if the necessary info was being logged (e.g. when I check an individual's user right log, it doesn't show when someone becomes autoconfirmed). It'd be great to see it if possible--I'd definitely be interested to know if we can confirm new users' creations are really a small part of the backlog, rather than these only appearing to be a small part of the backlog because the authors later made a few more edits and their contributions turned from blue to green in the graph. Thanks for the efforts to pin it down! Innisfree987 (talk) 04:13, 2 June 2017 (UTC)

Question on numbers

Tracked in Phabricator
Task T166269

First, let me say thank you for taking the time to write this and provide us with some numbers. From my first glance through it appears to me that the report is missing what is one of the more important numbers in the discussion: the number of pages created within the last 90 days that have been deleted, and then breaking that out by pages created by autoconfirmed and pages created by non-autoconfirmed. I'm still digesting the report and intend to read it again before making any substantial comments, but I do think that these numbers are critical to the conversation moving forward. TonyBallioni (talk) 23:53, 31 May 2017 (UTC)

That's a really good question. I'll see what we can find out, and I'll get back to you (or somebody will) soon. DannyH (WMF) (talk) 01:03, 1 June 2017 (UTC)

@TonyBallioni: We have an outstanding Phabricator task for getting numbers on that: T166269. Hopefully, we should have an answer by next week. Kaldari (talk) 20:32, 1 June 2017 (UTC)

I just want to say, in case someone makes the same assumption I did: I know it seems like these numbers should be easy to get. But the Vienna Hackathon was a real eye-opener for me on that score. I worked alongside numerous very skilled technical people who were trying to get this and other numbers, with unsatisfactory results. So thanks for your patience on this. JMatazzoni (WMF) (talk) 21:06, 1 June 2017 (UTC)

Rewrite the lede

Thanks for this analysis. Your data runs counter to what I assumed the problem has been. I still support WP:ACTRIAL but I also think most people shouldn't be editing Wikipedia, either. I don't think the best solution to "Time-Consuming Judgment Calls" is to let them age-off the list. I understand the logic presented but NPP is our sorting mechanism for problematic content. To my mind it would make more sense to have the "Time-Consuming Judgment Call" articles nominated by a bot (like WP:G13) for deletion after 60 days so that WP:DELSORT can assemble the subject-specific experts to make a determination. Orphaned, dead-end content (if not addressed by NPP) could remain out of sight for years until finally addressed. To that end, I think it advisable to re-write the lede to not include your conclusion, as reading that up-front almost blew all of my buy-in and I was about to write another dismissive denouncement of all of you in San Francisco. I'm glad I read the piece to the end and your presentation makes some difference as to how we proceed, but I could barely stomach that conclusion. Please bury it if you're going to provide it. Chris Troutman (talk) 00:41, 1 June 2017 (UTC)

Well, I'm glad you kept reading, and that we're talking about it. You bring up some fundamental questions about not just NPP, but about how Wikipedia as a whole looks at new content: Do we assume that new content is bad until specifically approved, or that it's okay unless it gets challenged? If marginal content remains out of sight until someone finds it and improves it, does that harm the encyclopedia? It sounds like your view on that is different from mine, but it's good for us to see that, and have those conversations -- not just you and me, but everybody who's concerned about it.

That being said, I don't know much about DELSORT, or how that workflow works. How active are they, and what does their backlog look like? DannyH (WMF) (talk) 01:00, 1 June 2017 (UTC)

DELSORT is a system of pages that list AfD discussions about a given topic. For example, WP:VG/D lists video game-related ones. (These pages are listed at WP:DST.) Editors look at new AfDs, evaluate which topics it's related to, and add them to the relevant topic pages. The point is to get input on these AfDs from people looking at their topic's DELSORT page. (I wrote a popular script that simplifies the tagging process to a dropdown menu.) I'm not sure if we've analyzed any data on the effectiveness of DELSORT, but I assume it's pretty good at getting topic experts involved in AfDs. Enterprisey (talk!) 01:38, 1 June 2017 (UTC)

(edit conflict) @DannyH (WMF): DELSORT doesn't declare that they have a backlog but the page itself claims that AfD (which they support) has grown to mammoth proportions. AfD has also made PROD'ding easier which intones that there are too many cases where only a couple editors !vote. I don't think DELSORT would be too happy about an influx of nominees but AfD has editors that watch specific topics and could render judgement about subject-specific notability criterion. Although most fans of a particular topic (the likely responders at AfD) will probably tend to be inclusionist (resulting in a lot of keep consensuses) that's the crowd that would be most likely to improve said articles.

I think Wikipedia contains a lot of content that fails notability criteria often as the result of POV forks, promotional efforts, and fandom. It does harm the encyclopedia to assume the "Time-Consuming Judgment Call" articles are ok because that substandard content pops up in search engine results and is visible to the reader. I also see little value in retaining poorly-written articles, regardless of content, especially where they rob a serious editor of a future four award. The assumption that the aggregate will crowd-source a good encyclopedia ignores the motivations of editors, which is why I'm pessimistic about new content, generally speaking. Chris Troutman (talk) 01:44, 1 June 2017 (UTC)

I do tend to think AfD would be completely overrun if however-many-thousand unreviewed entries began falling over the waterfall into the AfD queue; but, I'd be very much in support of a DELSORT-like tool within NPP, to help connect experts to subject matter as the report suggests. Being able to filter the queue by category or WikiProject would be one approach. (Not every submission initially has categories or WikiProjects but those are very easy to identify and add, in contrast, as the report notes, to the onerous and legitimately difficult work of the full reviewing checklist.) I know I'd do a lot more reviewing--and actual improving of the relevant entries--if I could easily find "Time-Consuming Judgment Calls" related to my interests and expertise, instead of going through them at random, which does quickly become wearisome. Innisfree987 (talk) 02:06, 1 June 2017 (UTC)

I'd definitely oppose a fall-off date (see Pathways To Peace a G12 candidate that survived 7 years as an example of what we could be missing if we had an arbitrary cutoff). I also think that the broader philosophical question asked about the nature of new pages actually has an empirical side to it as I mentioned above: how many new pages are deleted with 90 days.

I suspect that number is not insignificant, and it doesn't even have to be a majority to be harmful to the encyclopedia if they fall off to not be touched until someone realizes that it was a cleverly designed attack page or a copyright violation years later. The deletion numbers are critical to this conversation because they shed light on what the community has judged not to be acceptable from the recently created pages. TonyBallioni (talk) 02:33, 1 June 2017 (UTC)

Happily, I think the copyright violations are now handled by another workflow -- CopyPatrol has a bot checking every new revision, and comparing it to a search database to find potential copyvio. Each case is checked by a human volunteer, and either deleted, fixed or marked as a false positive. Community Tech built a new interface for it last year, and there's a small but super effective team of people who clear all the cases within about a day. So the Pathways to Peace-style copyvio should already be caught by CopyPatrol. DannyH (WMF) (talk) 03:19, 1 June 2017 (UTC)

Question

I guess my question, statistically, is whether there is some "critical mass" of views by reviewers that means the the page should probably be "semi-auto-reviewed" because enough people have passed on it, that it is likely to not be nominated for deletion, likely to survive that discussion, and/or likely to survive long enough for someone to find it and improve it? If that is identifiable, it seems preferable to an arbitrary limit like 30 or 60 days. TimothyJosephWood 02:17, 1 June 2017 (UTC)

As a follow up, if this question isn't answerable, and I suspect it may not be, would it be possible to incorporate a "pass" into the curator, so that we might be able to have an option for a good-faith "I'm not 100% comfortable with the subject/subject area/language/ect, and I'm not reviewing, but I'm not taking it to XFD/PROD/CSD" and try to gauge at what point enough "passes" might suffice to remove it from the backlog? TimothyJosephWood 02:44, 1 June 2017 (UTC)

We tried pulling some stats on clickthroughs from Special:NewPages and Special:NewPagesFeed and looked for the 25 example pages from the report, but I didn't include it because we're not sure that it's catching everything. There may be some clicks from tools like Twinkle that aren't counted. Some pages do get lots of clicks -- Heath Hitler got 13 clicks from NewPagesFeed, go figure :) -- but we didn't dig into it very far.

I think that building in some kind of staged system could be a promising direction. We were surprised to find that 28% of articles had been edited (not just viewed) by reviewers without being marked as reviewed. That seems like wasted work -- there should be more signals built into the system than just "reviewed" or "not reviewed". A reviewer looking at a new page should at least be able to help triage the page into some kind of group, or some kind of pass system like you suggested. DannyH (WMF) (talk) 03:15, 1 June 2017 (UTC)

Having slept on this, and with a strong cup of coffee, if something like this is ever tried, it should probably be a passive, rather than an active feature, and whether the results are collected in a publicly available log/other area of high visibility should also be carefully considered.

I expect that if you force a reviewer to actively push a "pass" button, or have some comparatively highly visible this user passed on this article, we are likely to have that read in the minds of some, this user was too damned lazy to do anything about it, rather than this user did some level of evaluation, and if the article was obviously toxic probably would have taken some action, and at some level, enough "meh" probably amounts to the article not being an "important" part of the backlog, and instead just something that can just as well sit around for a few months before someone starts improving it.

If you don't build in some lower level of personal accountability (for lack of a better term), then all its probably going to end up doing is renaming a second "review" button, that's going to suffer from the same "badge of approval" problem we currently have. Instead, the goal would be more along the lines of reviewing being a sign of "does some action need to be taken", and sufficient "passes" an indication of "does some action need to be taken right now". TimothyJosephWood 12:36, 1 June 2017 (UTC)

The "Pass" option could actually be useful if it would ask "why?". Then the reviewer would tag the page with "Needs a subject specialist reviewer, refer to WikiProject Rocket Science" which feeds the page into a sorting system that could be based on either the deletion sorting or stub sorting systems. This would also address the issue of most reviewers being "generalists" by passing the buck to relevant topic specialists. My experience is from AFC rather than NPP. At AFC it's a common practice to post a request for help to a relevant WikiProject. However their responsiveness is highly variable, in my experience the science projects are good: Physics, Chemistry and Medicine in particular respond quickly; others, particularly non-Anglophone country projects, hardly ever respond at all. Roger (Dodger67) (talk) 13:26, 3 June 2017 (UTC)

@Dodger67: lol. Thats what happened with me once recently, I dont remember the article though. (I got PCR user-right like a week ago.) I was sure the article met the notability criteria, but it was in a very bad condition (grammatical errors, confusing content, lack of refs and few others). So simply googled for related wiki project, and posted that article's link over there. In next 36-48 hours, the bad shaped stub article became a good "start class" article. The issue you are referring to, can be handled easily with this method. So much for "Time Consuming Judgment Call", and "generalist reviewers".
I mean, for these generalists, it wouldnt be much difficult to post a message to related wikiproject, or doing something to improve the article without getting much involved. —usernamekiran(talk) 15:05, 3 June 2017 (UTC)

Review of the report

The important thing to note is that adding more reviewers to this system will not make it work better.

This is true, and the community has not made a conscious drive to increase this number. However, the statement is true for the wrong reason(s). What does appear to be clear, although no one has been prepared to provide the stats, is that if each Reviewer had made only 50 reviews, the backlog would have long since been cleared. It’s caused by other patrollers who are not suitably experienced but who on the community's own insistence, are still allowed to tag pages.This naturally arouses the legitimate concern that some of the rights holders might possibly be hat collectors. Maintenance areas are a magnet to new and younger users - this is a fact of Internet life as anyone who has managed a busy web forum knows without having to spend months on dedicated research. The admins at WP:PERM are of course obliged to attach good faith to each application and only those that are clearly not suitable are declined. On this , I would invite Beeblebrox, Swarm, Juliancolton, Xaosflux, User:Mz7,Ymblanter, Lord Roem, and BU Rob13 to comment.

Backlog May 2017. Graph: Kudpung

• To be effective and present a realistic picture, the graph needs to go back to mid 2016 when the the backlog suddenly began to increase at an alarming rate, and for which everyone is apparently reluctant to provide a reason or even a theory.
  
• The WMF graph is also missing at least two other essential curves: The number of articles deleted, and the number of articles by new users that were deleted. The graph on the right shows the increase in the backlog since mid 2016.

While removing pages created by non-autoconfirmed users would reduce the burden on that first wave of reviewers, it would result in the loss of many potential good articles. It would also send a clear message to new Wikipedia editors that their contributions aren't wanted, potentially stunting the growth of the editing community.

This is based, apparently, on an opinion rather than on on fact, and nobody (from the volunteer community) has been suggesting pages created by non autoconfirmed users should be removed. If it's intended as an allusion to WP:ACTRIAL, then it's false. Indeed, It would not send such a message - in fact it would greatly reduce the number of totally inappropriate page creations (I won't even call them 'articles'). It would give good faith users time to read the instructions and polish up their articles to a state that would require much less intervention by other users who patrol and clean up; it would inspire other editors to provide help. We need to differentiate between regular editors who are happy to expand interesting potential articles, and the users who patrol the new pages to remove the unwanted ones and/or encourage the good faith creators to seek help. We cannot however, force our volunteers to do one, or the other, and certainly not both.

The top of the New Pages Feed says, in bold letters, Rather than speed, quality and depth of patrolling and the use of correct CSD criteria are essential to good reviewing.

The entire purpose of that piece of advice, which is misunderstood by the author(s) of the report, is specifically to avoid pages with potential, from good faith editors, from being flagged for deletion. This is one of the first major contradictions in the report.

A reviewer who doesn't spend enough quality time on a given review risks being blocked from reviewing...

This is conjecture and is inaccurate. The worst thing that can happen is that a New Page Reviewer can lose their flag and have to go back to patrolling without the convenience and advantages of the New Pages Feed and the right to mark pages as 'patroled'. As far as I know, of the 400+ reviewers, only two have had their flag removed and one of these was one who slipped through the grandfathering net. What does happen is, that very frequently newbies and other totally inexperienced users who are patrolling pages and getting it wrong are asked to refrain from tagging new pages until they have acquired sufficient experience. In these cases, the New Pages Feed serves as a double safety, and as far as I know, none have ever been blocked, and most of them were grateful for the advice they were given. We don't bite, but our tone may become decidedly less friendly if in spite of that advice the community has to issue a topic ban from patrolling new pages.

Summary

The urgent priorities are to:

• Continue to improve the New Pages Feed and its Curation Tool as requested at Wikipedia:Page Curation/Suggested improvements. This would ultimately make the system more appealing to New Page Reviewers who are still refusing to use it.
• Develop a proper landing page (please see this link) on the lines begun by Jorm for new users who are about to post a new page as their first edit. This would not discourage good faith editors. It would discourage spammers, vandals, hoaxers, etc., whose creations clearly make up the vast majority of new pages created by new users. The landing page was supposed to have been the other half of the new, New Page Patrol project. Instead, 6 years later we have an insurmountable backlog, a largely ineffective team of 400+ reviewers, and dozens of new users patrolling pages and scaring new, new article creators away.
• Combine the activities of AfC and NPR into one interface, for reasons not on this immediate agenda, but which DGG can probably explain better than I, but it would bring about a better utilisation of the recently created Draft namespace that was initially created as a place where new users could complete their articles withuot fear of rampant deletion before they are published in mainspace
• Prevent new and inexperienced users from doing certain maintenance tasks - this is not what was meant when the mantra The encyclopedia anyone can edit was coined.
• When using graphs to illustrate an argument, the full picture should be presented.
• It is not fully clear in the report that the difference between New Page Reviewers and ordinary patrollers is fully understood by the WMF team.

Conclusion

While I am pleased that the Foundation has finally decided to at least do some preliminary review of what is wrong with our page patrolling system, its taken a very long time, and I am sure that this progress has been achieved partly, though not entirely, through my constant whinging, lobbying, Skype conferences, and personal meetings with the WMF for a year with staff such as Danny Horn, Ryan Kaldari, MusikAnimal, Jonathan Morgan, Aaron Halfaker, Nick Wilson, and recently with Wes Moran (whose name has recently disappeared from the staff list). I would particularly like to thank Kaldari and MusikAnimal who have demonstrated the greatest understanding of the critical situation and have helped where they could within the limitations of their employment while dividing their time with their volunteer activities. Kudpung กุดผึ้ง (talk) 04:31, 1 June 2017 (UTC)

Discussion

• (moved from inline above)Regarding less experienced patrollers/hat collectors:Even if they make one valid patrol a year they helped the backlog. Not patrolling has them same affect as not being able to patrol, I don't see it as the cause of the backlog. However, if editors are making "bad" patrols that could be prevented by additional vetting before gaining access to the patrol tools, identifying additional granting guidelines can easily be incorporated to the permissions request system. — xaosflux ^Talk 11:03, 1 June 2017 (UTC)
• Going to create this section just to give us a place to discuss this. I agree with Xaosflux's inline comment above. Hat-collectors have a weakly positive effect – weakly in the sense that it may be zero, but positive in the sense that it cannot be negative. If we took away their shiny user rights, it wouldn't help. We have to face the reality that the problem is simple yet not easily fixed. The number of patrols that active editors are currently performing is exceeded by the number of articles being created. There's a high level of interest in improving the UI, talking about the problem, etc etc, but not much interest in devoting the actual manpower to fixing it. Or perhaps that manpower just doesn't exist. We need to stop talking so exclusively about the technical aspects of patrolling - who has the user right, the state of the UI, etc. - and focus on increasing the number of hours spent on patrolling. What are the segments of the editing community we haven't tapped into yet, and how could we more effectively get them on board? Chief among them are our content creators, who already have the most difficult-to-obtain skill required for patrolling, which is the knowledge of our content policies. ~ Rob₁₃^Talk 12:08, 1 June 2017 (UTC)
• I'd also like to separately urge admins to hand out the autopatrolled right whenever appropriate and editors to nominate others for autopatrolled (you can do that!) if you notice that someone's creating more than one article recently and qualifies for the user right. This helps the backlog in big ways! If we could get a bot to develop a list of editors who have created more than 10 articles in the past year, more than 25 total, and don't have autopatrolled (vs. the useless list of non-autopatrolled editors by total articles created, most of which are old editors), that would be helpful. ~ Rob₁₃^Talk 12:11, 1 June 2017 (UTC)
  • Rob, WMF has provided us with a report based on people who may qualify and have created an article in the last month Wikipedia:Database reports/Editors eligible for Autopatrol privilege. TonyBallioni (talk) 12:22, 1 June 2017 (UTC)
    • Hadn't gotten that far into this report yet. In two weeks, after I'm out of prelim hell (hopefully, anyway), I'll go through that database report and hand out autopatrolled as appropriate. ~ Rob₁₃^Talk 12:27, 1 June 2017 (UTC)
    • Please note, this is people that may qualify. — xaosflux ^Talk 12:56, 1 June 2017 (UTC)
      • Thanks for pointing that out. Yes, that is what I meant. Updated my wording. TonyBallioni (talk) 13:58, 1 June 2017 (UTC)

Kudpung, I am glad to hear you are passionate about this topic. I am curious though, about what you mean when you refer to newbies. New to Wikipedia does not mean someone is not capable. We all come from different experiences, be they educational, professional, or personal. No one (yet) has been born using Wikipedia. While there is a learning curve to Wikipedia, new people can also be capable people. We should not discount volunteers because they are new. Also, "spam, hoaxes, attacks, and other rubbish" are generally things that can be snuffed out using the duck test. No, it's not perfect, and it is not the job of a few, but of the community. If something can't be determined to be "spam, hoaxes, attacks, and other rubbish" in a specified time frame, let it loose for the community come together to examine it.

I am concerned because what I am hearing from your responses lead me to think this is almost like a peer reviewed journal to you. Many articles will be starts, and that's ok. If those starts didn't exist at all, maybe they wouldn't have the potential to grow into quality articles, and that person, place, or thing wouldn't get written about again for...(I know you like numbers, and I have none because this is hypothetical, so I'll just say)...well, maybe a long while. I worry also that this standard I am interpreting from your responses might not allow articles through, where they don't need to be blocked from opportunity, but given that opportunity. The same goes for new editors and volunteers.

Our actions do have human consequences. This is not just text on a page, but text that leads to someone on the other side of the computer/tablet/device, perhaps genuinely trying. Not all editors are bad. Not all newbies are inept. Being a linguist, I am sure you can appreciate nuance in language. I hope you can appreciate nuance in people. We need to remember good faith. It'll add years to your life. Jackiekoerner (talk) 04:11, 3 June 2017 (UTC)

Jackie, your worries are unfounded and I am absolutely not a deletionist per se - quite the contrary. I let a lot of stuff through that others wouldn't. When referring to newbies, there are people like yourself who make a conscious effort to read the instructions and get everything right, there are good faith new editors who don't read the instructions (a well known common Internet phenomenon) but whose contributions are in the main, with potential, hence they don't get it right (our fault for not presenting them with the instructions properly in the first place) but they might get some help from me, while those who deliberately come here to "spam, hoaxes, attacks, and other rubbish" are not volunteers. They get no help from me. In extreme cases I block them - naturally always within our rules. Kudpung กุดผึ้ง (talk) 08:10, 3 June 2017 (UTC)

• Having had the time to reread the WMF report and everyone's analysis, here are the initial thoughts I have:

1. The graph raises more questions than answers and doesn't match the text of the article. It doesn't show deletions, which is a huge oversight. It also seems to suggest that approximately zero-ten of the total pages created by new editors are reviewed a day if you assume the daily average of 78 is correct and the backlogged articles on the graph are in the 65-75 range the first two days after creation. This doesn't seem right based on my experience, and I assume deletions have something to do with it.
2. The 7% number makes no sense compared to the graph. If the daily percentage of the backlog that is created by new users is in the 20% range, this would be the daily decrease in the backlog. Additionally, if 7% is the correct number of new pages created by new users and their pages do account for more than double that proportion of the backlog this is a major issue.
3. The idea that an arbitrary fall-off date could be helpful is misguided in my opinion and runs contrary to the WMF's stated strategic goal of being the most trusted source of knowledge by 2030. Even if just 10% of the pages in the fall-off group had serious issues that slipped by this would be a major issue for Wikipedia. This also ignores the fact that we are in fact patrolling the end of the backlog, and while it is growing, the dates are moving up at the end. Removing these pages would prevent review and improvement for years for most pages. This might not be an issue with the majority of pages on the end, but for the ones where it is an issue, it could be a major one.

I'm very appreciative of the WMF for paying attention to this and engaging with us, but the main suggestion in this proposal (the fall-off) is a very bad one in my opinion. The data also needs a second look over because it seems to not be easily explainable and is missing some key numbers. I hope these observations add some value to the conversation. TonyBallioni (talk) 16:36, 1 June 2017 (UTC)

@TonyBallioni: There are some misunderstandings here regarding the graph.

Re 1.: No, there is no "huge oversight" regarding deleted pages. As explained above, the graph represents a snapshot of the existing backlog based on exactly the same data as that underlying Special:NewPagesFeed. I.e. it counts the pages that are still unreviewed at a specific point in time (May 25), with the time axis corresponding to the creation date (or equivalently the page's age in the backlog). Deleted pages are of course not part of this backlog - they have already been dealt with.

Re 2.: It does not make sense to compare the 7% number to the ratios visible in the graph. The former refers to the total number of articles created, but the graph plots the number of articles created on that day that are still unreviewed - which, needless to say, is different. (And e.g. the fact that autopatrolled creations never enter the backlog means that the percentage of articles created by non-autoconfirmed users will be higher in the backlog right from the start, even before any patroller has had a chance to look at them.)

You are of course right that deletion rates are interesting too, and I'm looking forward to working with you on these on the Phabricator ticket. But they address different questions. In particular, they will provide information about quality differences between articles created by autoconfirmed and non-autoconfirmed users - I think nobody will be surprised if the latter get deleted more often. But that is not the same as asking which group is contributing more to the backlog, or which sucks up more patroller effort by generating time-consuming judgment calls. For that, the backlog graph is more informative.

Regards, Tbayer (WMF) (talk) 04:40, 2 June 2017 (UTC)

Sorry if the oversight phrase came off harsh, I'm sure there was probably a better way of phrasing it. I still disagree here: the deleted pages are needed as a part of this graph because time that is spent reviewing them could be spent elsewhere. The number of pages deleted before they even hit one day out is significant piece of the backlog puzzle because of this.

In terms of the 7% figure, it makes sense, and I appreciated both your and Kaldari's explanations. I think my confusion is that by being presented alongside the backlog numbers it seemed as if that the report was saying that was the proportion of the backlog that would be reduced daily, which it is not. I think the daily backlog percentage is the more important number and would like to see that featured in the report if we can get it.Thank you for responding here. It is very much appreciated. TonyBallioni (talk) 05:25, 2 June 2017 (UTC)

Regarding verbiage: Thanks, and no worries!

"the deleted pages are needed as a part of this graph " - sorry, but then it wouldn't be about the backlog, but about something else. By definition, a backlog pertains to work not yet done (here, pages that have not been reviewed or deleted yet). And the focus on the backlog was not invented last week by WMF for this report: Kudpung and others have long been emphasizing it (as I'm sure you aware - but let me link just one example for the record and other readers' benefit), tracking its size etc., without including deleted pages. The graph represents an effort to better understand it: Instead of merely looking at one number (backlog size), we take a closer look at what kind of pages make up that total number. I guess that what you may have in mind goes more into the direction of tracking all review actions, which is interesting but separate.

"time that is spent reviewing [pages that are then deleted] could be spent elsewhere" - I get what you mean, and that surely true to some extent. But, as summarized in Danny's "time-consuming judgement call" concept, the time effort to review one page may not be exchangeable 1:1 with the time effort to review another page (and that's not even considering that they may require different patrol skill levels). Anecdotally, this should not be surprising: I think we can agree that a one-line school vandalism entry plausibly can cost orders of magnitude less time to review than a borderline notable promotional article submitted by a skillful paid editor. But the new data also provides some quantitative evidence indicating that such differences exist on a wider scale.

Regards, Tbayer (WMF) (talk) 06:46, 2 June 2017 (UTC)

@Kudpung: As mentioned, I am getting the impression that your criticism of the backlog graph is based on a misunderstanding about its time axis. The horizontal direction maps the creation date of the unreviewed pages, counting them at a fixed backlog snapshot time (May 25). In contrast, the graph you posted above (and others that track how the size of the backlog has been developing) varies the backlog snapshot time and does not distinguish unreviewed pages by creation date.

Thus, it doesn't make sense to demand that the chart "needs to go back to mid 2016 when the the backlog suddenly began to increase at an alarming rate". In fact, as can be seen from the raw data in the PAWS notebook (see the "queryresult" or "df_pivot" cells), or simply by selecting "unreviewed" and "sort by: oldest" in Special:NewPagesFeed, there are almost no pages in the current backlog dating from before December 2016, which was the pragmatic reason for starting the chart there.

Another way to look at this time axis distinction is that my backlog graph takes a specific value in your graph (for May 25, near its right end) and splits up that total backlog size by article age to get a more detailed understanding of what kind of pages made up the backlog at that point in time. One could even include both dimensions and make a three-dimensional plot: number of unreviewed pages (z-axis) per creation date (x-axis) and backlog snapshot date (y-axis).

Regards, Tbayer (WMF) (talk) 07:14, 2 June 2017 (UTC)

Non autoconfirmed new page creations

Wikipedia:New_pages_patrol/Analysis_and_proposal#Non-autoconfirmed_contributors appears to say that very few pages are created by non autoconfirmed users, but then it appears that autoconfirmed status is as measured now, not at the time of page creation. A quick look at the most recent creation tells me that non autoconfirmed new page creations are a lot higher than 7%. Autoconfirmation is a very low bar, and will be very easily met with a little fiddling of the first page. Just wondering about the facts. --SmokeyJoe (talk) 05:44, 1 June 2017 (UTC)

Well, after analyzing the graph further per the urging of Kudpung on my talk page, where I left a longer response I won't reproduce here, I have a question: where does the 7% number come from? By rough eyeballing of the first column in the graph it looks to be non-AC users account for ~75/275 of the daily backlog. That's 27%. That is a very significant number, especially considering how easy it is to reach AC. This is just eyeballing, so the actual percentage is likely less than that, but I would still but it in the 20% range from looking at it. TonyBallioni (talk) 06:34, 1 June 2017 (UTC)

There are a lot of things wrong with the graph and its interpretation. I'm hesitant to suggest it might be an attempt by the WMF to play down the crisis, but we have experts in data mining among the volunteer community who, given the actual requirement of what is needed to display the correct profile, could have done better. It is a great personal regret that I never learned to do these things and now it's too late. The correct data will certainly make some important changes to the focus of the report. Kudpung กุดผึ้ง (talk) 11:27, 1 June 2017 (UTC)

"There are a lot of things wrong with the graph ..." - with all due respect, my impression is that this assessment is based on a fundamental misunderstanding about the meaning of the time axis in the graph. I base this impression on your remarks above, e.g. the demand that "the graph needs to go back to mid 2016 when the the backlog suddenly began to increase" (I would be happy to extend it to the left beyond December 2016, but as detailed in the PAWS notebook, the timespan until November was excluded simply because the numbers there are basically zero and would not register in the plot).

I'll respond in more detail there. In case that won't help us to reach a shared understanding of what the graph means, a more concrete description of the alleged errors would be useful. Regards, Tbayer (WMF) (talk) 06:31, 2 June 2017 (UTC)

When it comes to promotional, sneaky promotional/POV articles, or WP:SNEAKY vandalism (either new articles, or in articles like Bay of Pigs invasion or Watergate Scandal); the perps seem to be aware of AC user-group. They usually wait for 5 days. Getting 5-10 edits in that time is not a big deal, they dont need to be article space either. Draft, userpage, talkpage; anything is good for that. I dont have the diffs to prove it, but i have observed it first handedly for a lot of times. Based on these observations, the conclusions/interpretations of the report/stats seem to be flawed. —usernamekiran(talk) 14:02, 1 June 2017 (UTC)

@SmokeyJoe and TonyBallioni: The 7% figure comes from https://phabricator.wikimedia.org/T149021#3287887. If you scroll down within the result, the full database query is provided. Let me see if I can explain this number a bit more and why it doesn't match the percentage of non-autoconfirmed articles in the backlog... 7% (or 6.57% if you want to be exact) is the percentage of new, main namespace pages that were created by non-autoconfirmed users from January 1, 2017 to April 30, 2017 that did not include the string "redir" or "Redir" in the edit summary (to try to screen out pages that were created as redirects). It includes pages that have since been deleted. This is the same query that gave us the "1,180 new articles created per day" stat. There are a couple reasons why this doesn't align with what you see in the backlog. First, the backlog excludes pages created by autopatrolled users. Second, the backlog includes pages that aren't technically "new", for example, pages moved into the main namespace from other namespaces like Draft and User, and pages that have been converted from redirects into articles. Unfortunately, it isn't possible to create a simple database query to match all of those cases, but you are correct that the percentage of pages entering the backlog that are created by non-autoconfirmed users is higher than 7%. The report itself says this in the second sentence of the Non-autoconfirmed contributors section. How much higher than 7%, we aren't sure. I think TonyBallioni's estimate of 20% is plausible though. One of the main points of the report, though, is that NPP is actually doing fine at patrolling articles by non-autoconfirmed users. Those pages tend to be handled pretty quickly (judging by the graph), especially the ones that are obviously bad. Hopefully we'll have better data soon on how many of those articles are actually getting deleted (T166269). Kaldari (talk) 21:24, 1 June 2017 (UTC)

To add, another important difference between the per-day ratios visible in the graph and the percentages being quoted here is that the graph - a snapshot of the backlog, i.e. still unreviewed pages - does of course not contain deleted pages, because they have already been dealt with and no longer need to be reviewed. (See also my somewhat more detailed remarks above.) Regards, Tbayer (WMF) (talk) 06:31, 2 June 2017 (UTC)

Tbayer (WMF), I'm sorry to say this, but from a practical viewpoint, that chart has not been very helpful. Besides the fact that I and several others have required further explanations to be able to understand it, the arguments put forward in its defence seem to be downplaying the crisis as not really needing any physical solutions. I know that that doesn't sound like a very good faith criticism, but because I'm only a simple linguist who spent 9 years at the TU in Berlin, I still don't understand complex charts. I'm absolutely sure that you are a highly skilled data analyst but not all of us are, and we're not able to see the picture the same way that a mathematiker does. On the other hand, my kiddy graph of the growth of the backlog is plain enough for everyone to understand and is the one that needs some of your expert analysis, especially of its strange shape. That would enable us, the volunteer community, to decide what we are going to do next, and if it needs more software, whether to write it ourselves, or insist that the Foundation allocates some resources to do it for us. Kudpung กุดผึ้ง (talk) 08:26, 2 June 2017 (UTC)

Kudpung, what would a revised graphic illustrate to you that you are not currently absorbing from the graph and the accompanying text? Perhaps I am the type of reader to use graphs as aids to the context. Perhaps you are more of a visual person. Regardless of what the graph looks like, this is a concern we need to discuss as a community, and in good faith. The numbers are still going to be the numbers, and the problem will still be alive and well, but sadly, more numbers won't equal a solution. Only we can all come together to make that. Maybe it won't be perfect at first, but something has to change. I think that much has been adequately illustrated. Jackiekoerner (talk) 01:39, 3 June 2017 (UTC)

I disagree with the analysis

Some parts of the the report are vague. At some instances, it is not clear if the report supports the reviewers/patrollers, or if it wants the right to be eliminated. Anyways. I mostly disagree with the section "This system is not sustainable". It says reviewers are generalists, this doesnt mean they dont have an area of expertise at all. If the reviewers are tripled in number, there are very high chances that at least one reviewer would be familiar with that particular field/category/subject with the backlogged article. Or there would be at least one editor from these 1000-1200 reviewers who would say "doesnt matter how long it would take, i will work on it". And in certain cases there is {{expert needed}} tag. I am pretty sure more than 95% reviewers know about these template as reviewer user-right is not granted easily.

"The only sustainable way to manage the backlog is to reinstitute the expiration date, which the system had from 2007 to 2012. An article that survives the gauntlet of reviewers for a reasonable amount of time – say, 30 days or 60 days – is unlikely to be picked up and fixed by a generalist new page reviewer. Pages that survive past that deadline should be improved by subject matter experts, which is the way that Wikipedia works. With a 30 day expiration, the backlog on May 30th, 2017 would have 5,650 pages instead of 21,800. With 60 day expiration, it would have 10,200 pages."

I think it is prohibited for subject matter experts to touch the page if it is unreviewed. Maybe they get blocked for 24 hours for just clicking the "edit" on an unreviewed page. Why not set the expiration date at 10 days? It will be really good for everybody. This will be exactly like, changing the definition/upper limit of high blood pressure to be able to say "no! The parient doesnt have a high blood pressure." —usernamekiran(talk) 13:50, June 1, 2017 (UTC)

How does adding an expiration date supposedly "fix" anything? Yes the backlog stops reporting so many numbers , but it doesn't change the quality or content of the new articles and doesn't impact readers or editors. We already have a 30 day INDEXING expiration - and that is something that is actually reader impacting. @Kudpung: please tell me I'm not missing something here! — xaosflux ^Talk 14:45, 1 June 2017 (UTC)

I absolutely support the reviewers/patrollers, who are doing really important work. What I'm saying is that the system that's evolved is unsustainable, for two reasons: the standard for an acceptable page/review is getting higher, and we're not allowing good pages to age out of the backlog naturally. That means that people are spending a lot of time improving pages that are already pretty good, rather than clearing out bad pages and allowing the pretty good ones to age out. The other side of the high blood pressure analogy is that the patient doesn't want to change any of the behaviors that led to having high blood pressure.

Xaosflux: the point of an expiration date is that it encourages reviewers to actually make decisions, rather than looking at the page, making an edit, and not marking it as reviewed. There are a lot of pages that have been looked at and even improved by multiple reviewers, but they're still in the backlog because there's no consequence to just leaving it there. In my opinion, that system will always create a backlog. DannyH (WMF) (talk) 15:15, 1 June 2017 (UTC)

@Xaosflux: yes, thats my exact point. I apologise for using sarcstic/rhetorical words, sorry for the confusion. I meant, not looking at something doesnt make it go away from existence. It still remains there. Thats what happens with the backlog, even if the expiration dates are changed, the bad articles which were not looked at will remain there. They are not going to change themselves in good articles; the copyrighted content is going remain copyrighted, promotional content is going to remain promotional no matter if the article is the backlogged group or not.

@DannyH (WMF): yes, I partially agree with you. The basic point of everything on wikipedia is, we should work towards creating good articles. Then, whether it is backlogged or not becomes sort of immaterial. If there are active reviewers, then at the least, the "easy ones" will get marked as patrolled/reviewed. That would in turn, result only the articles with "Time Consuming Judgment Calls" (TCJC). These articles can later be handled by the editors of that particular field. After weeding out the "easy ones", I dont think handling the articles with TCJC would be a difficult matter. (Yay! I just coined a new term lol).

In conclusion, we need reviewers who are active, and a good strategy. Strategy is the key here. —usernamekiran(talk) 15:42, 1 June 2017 (UTC)

(edit conflict) Yes, the point of an expiration date is that it encourages reviewers to actually make decisions, rather than looking at the page, making an edit, and not marking it as reviewed, DannyH (WMF), but I doubt that the rest of the first part of your comment is in any way founded on facts. I would point out (and for Xaosflux), that NO_INDEX actually expires after 90 days (and fortunately so) . And here's how it was done, by Kaldari, Roan Kattouw (WMF), Cenarium, and myself, in a perfect example of how true Community/WMF collaboration should ideally take place.

Since I stepped back in February from 6 years of actively campaigning for improvement in the way we police new pages, I have gone back to spending sometimes up to 14 hours a day reviewing new pages. Don't get me wrong - I'm not doing it specially to help reduce the backlog, although of course that happens; I'm doing it to do that all-important empirical study. The standard for an acceptable page/review is not getting higher. In fact it's pretty much the same as it ever was and this does not explain the backlog at all. Reviewers are not spending any more time on their reviews than they did before. To be sure of your facts you need to check on the Curation logs of every one of the New Page Reviewers and see how long they took over each review during a session.

There needs to be a clear understanding of the distinction between New Page Patrollers and New Page Reviewers. They are both very different animals. Partly the reason that the system is unsustainable is because Community Tech blocked any development that would have made New Pages Feed and its Curation Tool the perfect alternative to Twinkle, and partly because the community itself insisted on allowing MMORP gamers to continue to mess with New Page Patrol. Those of us like DGG and me who patrol new pages to gain an overview of the bigger picture, spend half our time chasing the newbies away from this important and complex task. I fail to understand why the Foundation would support a system that allows inexperienced users to bite good faith creators of new pages on the one hand, while suggesting that spam, hoaxes, attacks, and other rubbish should be allowed to 'to age out' for simply not having been patrolled within the time limit. Kudpung กุดผึ้ง (talk) 16:19, 1 June 2017 (UTC)

@DannyH (WMF): As a benefit of my career, I've gotten somewhat adept at differentiating between motivations that actually drive behavior and good-on-paper motivations that don't. Your suggested behavior pattern for editors sounds plausible, but I highly doubt our reviewers think about the overall impact on the backlog of their individual actions, since their individual actions form such a small part of the aggregate. I can't see any plausible method to shift from "If I don't do this review and spend time on another, someone else will just do this" to "If I don't do this, no-one will". If anything, we'd stress out people working on the backlog from oldest to newest, who would see things going by without any ability to really stop it. That could cause burn-out due to a sense of not accomplishing anything. If you want to say we should impose an expiration date because the content remaining in the queue is a time-sink that can be improved just as well organically, then I disagree, but that's a matter of opinion. If you're saying we should impose an expiration date because it will drive a higher quantity or quality of reviews, I think that's fundamentally incorrect. There's no evidence to support that. ~ Rob₁₃^Talk 00:12, 2 June 2017 (UTC)

@BU Rob13: Sorry, I should be more specific about where I think the motivation matters. I think having an expiration date would mean that the organizers of the team would be less likely to instruct people to slow down and leave hard pages to someone else. In September 2015, the description of the team's primary purpose changed from identifying pages for deletion to judging whether pages merit inclusion (diff), and the nutshell summary on the Article checklist page encouraged people to leave pages for another reviewer (diff). Those changes accelerated the process towards an unmanageable backlog. I think that an expiration date would help organizers focus on fixing the bad pages, and letting the pretty good pages age out of the backlog naturally. DannyH (WMF) (talk) 18:12, 2 June 2017 (UTC)

DannyH (WMF) Please define "organizers" and "team" as used in your post above. Roger (Dodger67) (talk) 13:53, 3 June 2017 (UTC)

Dodger67 I'm using the word "organizers" to mean the people who are empowered by the group to make changes to the system, and to enforce the rules. The clearest examples are the people who made the edits that I reference in the report, but it could also mean the people who tell other reviewers whether they're doing it well or not. DannyH (WMF) (talk) 20:00, 3 June 2017 (UTC)

Some Background

There seems to be a popular misconception that ACTRIAL was only about preventing non confirmed users from creating pages. That perception is completely wrong and discredits those who worked hard to create the ACTRIAL project and the hundreds of users who voted the overwhelming consensus for it. In 2006 by withdrawing the 'right' of IPs to create pages, the Foundation already acknowledged that Wikipedia is organic and that the rules occasionally need to be modified accordingly.

It is erroneous to allow the impression that the current backlog began concomitant with the creation of the New Page Reviewer user right in November. It didn't. This current backlog actually stretches back to mid 2016 (where it was 'only' 5,000). This was a already a grave concern and is what gave rise to the talks in Italy and the run up to the creation of the New Page Reviewer group in November. In fact for a while, until it suddenly started rising dramatically again in February, after the roll out the backlog actually began slowly but surely to diminish.

The Foundation has now given us a page full of comment, which is genuinely very welcome and highly encouraging and I'm sure that those commenting here will read it entirely and carefully, but in order to get properly up to speed, the WMF team should probably also be encouraged to do the community team the courtesy of reading the whole of this page: Wikipedia talk:The future of NPP and AfC. Kudpung กุดผึ้ง (talk) 15:10, 1 June 2017 (UTC)

Does this mean, "After 30 days, any pending submission passes by default"?

• Update: DISREGARD EVERYTHING I SAID. I had no idea what was going on. Blue Rasberry (talk) 23:59, 1 June 2017 (UTC)

There is a lot of information here and I am not following this proposal. Suppose that someone makes a submission and it gets no review after 30 days. After that point, will it move into Wikipedia mainspace in the same way as an article which passed review? Blue Rasberry (talk) 20:08, 1 June 2017 (UTC)

@Blue Rasberry: Not yet. But yes, it is being thought i think. "To help decrease the backlog" apparently. —usernamekiran(talk) 20:39, 1 June 2017 (UTC)

• previously pinged incorrect username. Bluerasberry —usernamekiran(talk) 20:40, 1 June 2017 (UTC)

I would like to see a random sample of proposed "default pass" AfC submissions go to AfD. I have imagined that the bar for passing AfC is the same as passing AfD. I do not see data to back the claim that the default for not getting a review should be "pass" instead of "delete". Deleting anything which does not pass a review after a certain amount of time is an option also. In AfD if something does not clearly pass, then people vote delete. The current AfC process does not currently default to delete in that way because it is intended to be friendlier to new users.

In this proposal, what I am seeing is the idea that multiple reviewers pass problem cases. I want those out of the queue to make way for better review, and I am not convinced that marginal cases are better passed than deleted. If marginal cases are 50% passing quality and 50% deleting quality, then I might say delete. Some of this proposal seems to suggest that marginal cases are 90%+ passing, which if that is the case, I can understand passing them all by default. If the numbers are 50% or lower, I say delete by default.

I wonder if changing the structure to pass by default will change the stability of tendency to pass/delete article submissions.

Overall, I still do not understand the data presented here or what it means. Blue Rasberry (talk) 20:56, 1 June 2017 (UTC)

I'm... confused. Where does AfC come into this? TimothyJosephWood 21:11, 1 June 2017 (UTC)

I think you mean "AfD". My claim was that the AfC process should never move a draft into the mainspace if that draft would immediately fail AfD. I expect that most people would oppose a plan to move articles likely to fail AfD into the mainspace. How do you feel about having articles likely to fail AfD in the mainspace? Blue Rasberry (talk) 23:50, 1 June 2017 (UTC)

Hi @Bluerasberry: The ultimate decisions about how the system should work are up to the New Pages Patrol team, and the volunteer community as a whole. The report that we posted is our view of what's causing the current situation, and recommendations for changes that NPP can make. It's meant as a step in a larger NPP/Foundation/community conversation.

The fact that there's a 22,000 page backlog means that there's something that's not working with this system. At the moment, the default setting is de facto "pass", because all of those pages are currently live on Wikipedia. As far as readers and editors are concerned, whether an article is marked as reviewed or not is invisible. The pages are part of the encyclopedia.

As I wrote in the proposal, I think that the main drivers of the huge backlog are the increasingly high standards for what constitutes a reviewed page, which changed in September 2015, and the tighter control over how fast reviewers are going, which changed in November 2016. The system is asking people to go slower and spend more time with each review, and encourages people to pass and move on to another page. That means reviewers pass on the most time-consuming pages, and those are the pages that end up in the backlog. Telling reviewers "just go make 50 reviews" is too much to ask, if each review could take days to complete. There needs to be something in the system that acts as a motivator for people to actually make decisions, whether that's pass or delete.

For the Foundation, we're very interested in helping this system to work better, but we don't want to invest resources in a system that we think is fundamentally unsustainable. We can't make the decisions about how the system works, but we want to be involved in the conversation. I hope that makes more sense? Let me know if I'm still being unclear. DannyH (WMF) (talk) 21:30, 1 June 2017 (UTC)

@DannyH (WMF):What do you mean by "all of those pages are currently live on Wikipedia"? Do you mean "live" as in "draft space"? I would not call that live, so clarify if that is what you mean. If that is the kind of default you mean - "still in draft space, but no longer in the review queue", then I was not understanding that. Blue Rasberry (talk) 23:48, 1 June 2017 (UTC)

Bluerasberry, the solution that you and Chris Troutman suggest (nominating all articles older than X days for deletion) is already technically possible. A bot could easily handle that. It might be worth seeing if there is broader consensus for implementing that as a solution. Kaldari (talk) 21:50, 1 June 2017 (UTC)

@Kaldari: nope, it would be a disastrous move. Plenty of good articles would get deleted just because a group of limited users couldnt process it. And bots are dumb after-all. Isnt that right, DumbBOT? (Just nod your head if you want more electricity.)

It is easy to foresee this solution will never get the consensus. —usernamekiran(talk) 22:08, 1 June 2017 (UTC)

It is easy to foresee this solution will never get the consensus. That's probably the one thing we can all agree on. TimothyJosephWood 23:27, 1 June 2017 (UTC)

@Timothyjosephwood and Usernamekiran: Can you restate your view? There are lots of article in limbo in draft space. The backlog is 20,000. Options for addressing this include leaving them in limbo/draft space perpetually, or deleting them after some time, or moving them from draft space to mainspace after some time. If I understand correctly, you are saying that there will never be consensus to delete draftspace articles after some time. Do either of you believe that there can ever be consensus to pass these articles from draftspace into mainspace by default after some time? Blue Rasberry (talk) 23:48, 1 June 2017 (UTC)

@Bluerasberry: All of the pages that we're talking about are already in mainspace; that's the 22,000 page backlog. Drafts/AfC is a different workflow. DannyH (WMF) (talk) 23:50, 1 June 2017 (UTC)

(edit conflict)@Bluerasberry: Erm... I think you were confused right before you posted your first comment today. We are talking about the articles that are already published, and need to be reviewed. —usernamekiran(talk) 23:55, 1 June 2017 (UTC)

Thanks, I was confused. Blue Rasberry (talk) 23:59, 1 June 2017 (UTC)

The WMF's solution is really a head-scratcher to me. They think the problem is "The queue is too big" so the solution is "Just throw stuff out of the queue after 30 days". If the only metric we're going by is length of queue, I have an even better solution. Get rid of the queue entirely and the queue length will be zero! The secondary metric (quality of content) is lacking in this report. Further, they fundamentally misunderstand the guidance given to patrollers. They wrote "Following the Article namespace checklist – the minimum effort that a reviewer is supposed to do – this article would probably take days to fix. You'd have to track down references, most of them not in English, and completely rewrite the page from scratch." That's incorrect. The guidelines for patrollers say to fix easy issues and tag more complicated things, which doesn't take long at all. Same goes for notability. If notability is seriously questionable, tag with questionable notability (or bring it to AfD for more opinions, which is also fine). Detecting issues and fixing issues are very different things with very different time commitments, and so the analysis is flawed. If we have a messaging problem where patrollers think we're asking them to make every article GA-quality, let's talk about that. But if every reviewer followed the guidance given to them, there's no reason to believe articles in need of improvement are driving the backlog. ~ Rob₁₃^Talk 00:06, 2 June 2017 (UTC)

if every reviewer followed the guidance given to them But they're not. So we can talk about a perfect world all day, but how do we square the de jure with the de facto in a way that in a reasonable logistical sense accomplishes the mission? TimothyJosephWood 00:15, 2 June 2017 (UTC)

I'm just going to add a touch of a rant so forgive me. I've seen if only every reviewer would review X amount of articles the system would work so many times it's starting to make me nauseous. They're not...we're not and we have to ignore the idea that it's a problem with the reviewers because that's not a problem we can solve. So what problem can we solve? TimothyJosephWood 00:20, 2 June 2017 (UTC)

Well, until we see the new numbers the WMF is working on for deletions and backlog by autoconfirmed status at the time of creation (not today) I don't think we know much. Like Chris troutman and I suspect others, I haven't seen anything that has convinced me that pulling the trigger on ACTRIAL wouldn't help. Others have also suggested in the past switching the default place for page creation to be draft space, which would shift the burden. What I am confident in is that the obscuring of the number of unreviewed pages by having a cutoff isn't the solution. TonyBallioni (talk) 00:29, 2 June 2017 (UTC)

• It seems quite clear (to me at least) that either now they've started the dialogue, the Foundation is not fully paying attention to what is being posted here by the community, or the Foundation team has never actually spent any significant time at the coal face - or both. It also seems to me, and I sincerely hope I'm wrong, that they are belittling the huge efforts made by a few concerned editors over the past 6 years to to get New Page Patrolling cleaned up and some genuine good faith attempts to bring some modernisation into into the way new pages (I won't say 'articles') are created and by whom.

We never had an explanation either as to why Jorm's development of a proper landing page was quietly, but deliberately shelved and hidden behind a wall of multiple page moves. Jorm won't tell us why, not even me when we met privately in 2014 (possibly because he's muzzled by some non-disclosure clause). Let's not forget that the lack of a proper landing page is half the current problem.

And all the while I am very curious why DannyH (WMF), who incidentally personally blocked further development on the Page Curation system by insisting that this critical issue join a community wish list, until suddenly opening this dialogue, is still insisting that this backlog is a recent phenomenon that dates back to when we rolled out the New Page Reviewer right. The table graph I posted above should be clear enough. I was politely given the brush off already in June last year in Esino Lario by a WMF staff when I tried - in the presence of other en.Wiki admins - to discuss the situation, and a 3-hour private discussion on a hotel terrace with another one during the same conference was constantly met with 'rest assured that the WMF is actively doing its best to resolve these issues' (or words to that effect) - these are the kind of comments we're used to hearing from British Prime Ministers. In the corridor of GWU in D.C. back in 2012 when we attempted to broach the issue we were actually told very rudely to F*** off, by a contractor, and his boss, a C-Leveler, told us to mind our own business. I don't like conspiracy theories, but what on earth is going on? Kudpung กุดผึ้ง (talk) 01:23, 2 June 2017 (UTC)

There's no conspiracy. Jorm's Article Creation Workflow was killed for two reasons: The community was moving towards creating a Drafts namespace (which would have required reworking the workflow), and the WMF decided that working on the Echo extension (i.e. notifications) was higher priority. The engineers that were working on Article Creation Workflow and Page Curation were simply re-assigned to work on Echo. Luckily we were able to (mostly) finish Page Curation and launch it (but only for English Wikipedia). Article Creation Workflow was never more than halfway finished, and at this point it would need to be redone from scratch. Kaldari (talk) 02:11, 2 June 2017 (UTC)

• On "pending submission passes by default", I'm not sure what this means. I thought this was about namespace 0, where there are no "submissions" and nothing ever "passes", because everything is always a work in progress. The patrolling features are about reducing redundant work: if somebody else has already patrolled a page/edit, I can skip checking it for obvious vandalism. --Nemo 13:11, 16 June 2017 (UTC)

What is wrong

It is completely wrong that "The only sustainable way to manage the backlog is to reinstitute the expiration date" The effect of a cut off date will be to keep articles from ever being reviewed. There is no reason why anything at all should totally drop off the end--if there is any purpose to reviewing, everything needs review. All that having an artificial cut off does it prevents us from realizing the extent of the backlog, by deceiving us into not seeing it. Having such a date is the basic recommendation of the report, and it is absolutely counterproductive, an admission of defeat. It is the reaction of bureaucrats who want to pretend they have control of a problem, not editors who know the limitation of Wikipedia. It will indeed make the system look better. Bureaucrats and system professionals care about this, they do not really care it it fulfills the function. They just want it to look professional. (This is not meant as personal--when i was a professional in a complex system, it was most important to me that my library appeared excellent --I was very aware of the things that were grossly imperfect but that I could not affect, and I did very well at making them invisible.)

It is completely wrong that "adding more reviewers to this system will not make it work better." The most important thing to note is the direct opposite: without adding more reviewers to the system , it will never work much better. The only real way of improving anything at WP is more participation in the process. The principal goal of us all should be to increase participation at every step--starting with people reading articles being willing to make obvious improvements, all the way through every step in increased involvement, all the way to being regularly writing new articles. If retention at each of the many steps were increase even slightly, the overall effect would be significant. When we look at the details, our goal should mostly be to remove impediments. We do not need to come from outside to design a system. The entire principle underlying the Wikipedia projects is that the system is self-designing and self-correcting. Not everything can be done by such methods; Wikipedia is not the all-encompassing intellectual product of mankind, but has a special role: a general purpose encyclopedia of first resort universally available. What can be done by amateurs working with informal coordination is what we should do. What requires specialists or professionals or centralization is what should not be part of the projects, and was never from the first intended to be.

It does not take expertise to do most reviewing--neither great expertise at WP and certainly not expertise in the subject. That doesn't mean one person can do everything, but as a person does get experience, they can move to the more difficult articles. Reviewing is meant to be a first pass, and part of the problem is that its function and role has become overloaded to the point where it becomes an impediment. It should not be reviewing in detail or definitively; if it were, it would indeed be impossible to keep up. Rather, the point of reviewing is a first pass, to mark the things that must be removed, and to indicate some of the key problems. Articles are further assessed continually as people see and work on them. Each of us does have subject limitations, but again , the basic principle of a project with widespread participation, is that among us all, we will cover all the fields. What cannot be done this way, is unsuitable for us to attempt. We've seen in the development of WP, a wider and wider expansion of what volunteers are able to do and want to do. It has not been centrally developed.

Given the existing or attainable levels and types of participation, we do not primarily need improved technologies of review--I and I think most good reviewers almost never use the reviewing toolbar except for its convenient functionality in scanning. (I do use twinkle--this is an example of a combination of locally developed stopgap methods whose usefulness has been greatly expanded by widespread adoption, rather than something actually planned from the first.) What we do need is for most experienced WPedians regardless of primary interest to look at a small number of new articles each day,as part of their normal participation here. Accepting the figure of 1200 articles a day, it will be better if 200 people each review 6, than if 20 people each review 60.

There is a genuine but limited role for professionals at WP: to devise tools that volunteers seem not to want to deal with. There are two tools we need (undoubtedly others can suggest additional ones): a prescreening for likely copyvio at the time of submission, and a rough system of subject classification at input. The available technology can do these. So can AI, but we need not wait for that. Where AI might be useful is in distinguish promotional edits, where we cannot yet explicitly specify how we are judging.

Nothing about this is actually broken, in the sense of not working. Many things are not working very well, and if WP is to do what WP can do, that will always be the case. Our role is to work at the frontiers of what volunteers can accomplish. Things will always be rather rough out there. It's supposed to be that way. It's for doing this sort of unpredictable work that we need our sort of project. (I want to emphasise that I know individualy about half the people contributing to this report--based on what I know of them, any one of them could have done more realistically by themselves,and all of them do more realistically in their volunteer capacities.) DGG ( talk ) 03:37, 2 June 2017 (UTC)

@DGG: I'm really not seeing an expiration date as a way of masking a problem. I'm seeing it as a way of acknowledging what's actually happening with these articles -- they're pages of the encyclopedia. You can't tell that there's anything different or wrong about these pages, unless you look at NewPagesFeed and scroll for a while. The other function of an expiration date would be to help people focus on making decisions, rather than collecting pages in a backlog that nobody wants to review.

I think my question is: how would you explain where the 22,000 page backlog comes from? According to the chart posted above, the backlog has jumped from around 7,000 to 22,000 in less than a year. The NPP system changed quite a bit in the last two years, as I wrote in the report. I don't think that any external changes have impacted on the system. So if this approach is the best one, then why is there a backlog now? DannyH (WMF) (talk) 17:19, 2 June 2017 (UTC)

No DannyH (WMF), the read of my clear kiddy chart is not quite as simplistic as that and while it clearly does not appear to correlate with your hypothesis, it's more recent and possibly dependent on totally different stimuli. For one thing, it has an extremely odd shape. Please help to explain that for us because nobody among the volunteer community can/will. Kudpung กุดผึ้ง (talk) 06:27, 3 June 2017 (UTC)

Kudpung, what's the source of those numbers? I can't tell from the chart. Can you go back further than July 2016? I'd be interested to see the impact of the changes made in 2015, if there were any.

For the odd shape: I don't think there was any unusual spike or decline in the number of pages created over the last year, besides the usual seasonal changes related to holidays and the school year. Assuming that there was a more-or-less stable number of pages created, then the chart represents the number of reviews being done at those times.

I know that one reviewer, SwisterTwister, was reprimanded and kicked out of the group sometime around spring-to-summer 2016, for reviewing articles too quickly and not doing a good enough job with them. I've browsed some of the AN/I threads about it, but I don't know the exact dates. Taking SwisterTwister out of the group, and possibly other people for similar reasons, could result in the rise in the backlog from spring 2016 through the fall.

The discussions around creating the user right happened in the fall, and the user right change went into effect in November. It's possible that those discussions, and the process of deciding who deserves to have the user right, got people motivated to do more reviews than usual. When someone has to apply for a user right, they're excited when it's granted, and they probably want to do a lot of work, to prove that they really deserved it. That could explain the stable period that happened in the winter.

I haven't followed the more recent discussions on the NPP pages closely enough to know what the group was doing over the last few months, which could explain the drop and rise in the backlog from February to May. Kudpung, I think that I saw on some talk page comments that you had stepped away from organizing the group in the last few months. Maybe that had an impact on the number of reviews that you or other reviewers were making?

Anyway, my point is that the rises and drops in the chart are related to internal changes in the way that NPP works, who's participating, and how motivated they are to review pages and mark them as reviewed. As far as I can tell, the rises and drops don't have anything to do with the number of pages that were created every day. So the rise of the backlog from around 6,000 to 22,000 in a year means that the current system is not working well. The only way to get the backlog under control is to change the current system. DannyH (WMF) (talk) 20:33, 3 June 2017 (UTC)

DannyH (WMF), in Hong Kong 2013 Erik Möller asked me at length for feedback on how the Curation tools were performing and what the Foundation could do to improve themI was left beieveing tose suggestions would be carried. My main concern for stepping up interest again in early 2016 in NPP and getting the bugs ironed out of the Feed /Curation tool were due to the increase in the poor patrolling by new and/or inexperienced users, and even more importantly the significant change in the profile of the influx of new articles - i.e the massive increase in organised paid spamming of the encyclopedia in August 2015, on a really huge scale which the WMF is not aware of but which we the volunteers had to clean up such oranised spamming which totally overwhelms our already stretched resources, making credible arguments on the one hand for suitably qualified patrollers, and for a relaxation in the rules governing our strictly limited use of the CU tool on the other. The backlog had been ticking over at around 5,000 for years until it dramatically began to rise immediately following Wikimania (where my intended presentation on how to better use the Curation toolset to identify and combat subtle forms of spam and COPYVIO was at the last minute renamed, refocused, and presented instead by the WMF, so that can't be the reason either

The stats for the graph are taken from the footer at the New Pages Feed.

There could possibly be some correlation between the shape of the curve and the date of creation of the empty improvements list on September 2, 2016‎ by Noyster and which was rapidly populated by me.[1]. Around the time of the roll out of Reviewer, there was a flurry of activity to successfully address some of the requests, but when activity dropped off due to the Foundation declaring other community wishes to be of greater importance, by the time it became clear in February that there was going to no further development on the Page feed/curation interfaces, interest in using them and patrolling new pages dropped off. Also at that time in February, the promised Barnstars were never issued by the NPR coordinators. Some of us might not be so interested in bling, but judging by the offer of trophies for good work on their drives, other sub-projects, such as WP:GOCE, seem to thrive on the encouragement their rewards bring about.

At the same time, I do not believe there is any correlation between the sudden reprise of the steep increase in the backlog and my February departure from physically maintaining and motivating the NPP project for the last 6 years - I think it’s pure coincidence. As Sadads correctly points out, reward is the prime motivation, by which he means that if we can reward the community with the improvements they are asking for to Page Curation, they will use it and love it. I use it religiously and in spite of it being a WMF development, I think it’s one of the best things that ever came out of SF - certainly far more important than say, notations. We do need to find out what exactly the influences or events are that caused those extremely marked changes in the pattern of the graph. If we can put our finger on that we may be getting closer to a solution. I think that is more important than making it sound as if you regret Page Curation being developed in the first place - let's not throw our toys out of the pram just yet, it's a bad workman who blames his tools. Kudpung กุดผึ้ง (talk) 23:26, 3 June 2017 (UTC)

(Might be a bit nitpicky) @Kudpung: But you actually misinterpret what I mean by reward: rather than expecting a reward from WMF for the work we do (we choose to create this queue because we see some value in it -- and we are volunteers creating the governance strucutures by which we generate quality -- so there is no reason we should expect support from WMF on a self-created problem), I am arguing that rescuing or engaging with new content can be a reward in and of itself, if and only if, I can review content that I understand is valuable to our larger mission. The filters I am proposing, would help folks find the intrinsic rewards in this work, just like how folks like you, DGG or myself, find value in seeing content succeed because of our existing workflows. Backlogs are rewarding, because they indicate that something indicates that work on that backlog is of value -- however, by throwing all of our new content into one backlog, without subdividing it more -- its really unclear to many editors why its important to take items out of the backlog. Sadads (talk) 14:03, 5 June 2017 (UTC)

we need to review new submissions. This is the sort of activity which is better done in an orderly fashion. The current method is not completely successful in getting everythign looked at, but it's better than having no method at all. Though enough people even with this method will quite possible deal with it, we need some positive suggestions for improvements. A number have been madde, and noe adopted. e need to discss them on their merits one at a time--there is nothing to be gained by tearingapart what we have unless we have something better. DGG ( talk ) 23:53, 4 June 2017 (UTC)

Fair enough. It seems like people are coming up with a lot of good ideas right now, which is great. DannyH (WMF) (talk) 14:44, 5 June 2017 (UTC)

The backlog is not a self-created problem, Sadads. Page Curation has been doing (more or less) alright during the 5 years of its creation until its inadequacies, (or more accurately, those of the entire NPP system) were exposed by the new trend in 2015 that has shaped the profile of the envelope of what we call the fire-hose of totally inappropriate new 'articles'.

’’So the rise of the backlog from around 6,000 to 22,000 in a year means that the current system is not working well. The only way to get the backlog under control is to change the current system. ‘’ - well, not quite: The only way to get the backlog under control is to improve the current system. And, very importantly, introduce some stricter control over the type of new articles before they get created: prevent the rubbish, while encouraging good faith users to better read the instructions and follow the guidelines when creating content that might just be acceptable.

I didn't misunderstand your meaning of 'reward' for an instant. I used the term 'reward' metaphorically - the volunteers are not looking for handouts from the WMF in return for good work. The volunteer community expects the WMF to provide the tools they need in order to uphold the values the WMF insists the volunteers maintain.

One of the problems is that it is not always easy to understand the anatomy of an organisation that is largely self-managed by a large group of unpaid volunteers. Let’s not be confused with the role of the volunteer community here. It’s comprised, very broadly of three elements:

• Those with subject knowledge who add respectable, encyclopedic articles and/or expand the content
• Those who responsibly get rid of the unwanted content and keep the content free of disruption by others
• Those who repair and improve substandard but nevertheless encyclopedic content and provide help to the new and/or inexperienced users who show the potential of becoming permanent and positive members of the Wikipedia community.

Stick and carrot tactics are not going to get the community to do better work of building and and keeping clean the content of this encyclopedia. What we need right now is for the expert statisticians and data analysts employed by the WMF to explain to us why that simple but very worrying graph of mine has such a strange shape - to prove or disprove once and for all any correlation between its shape and any events on the ground. When we have that, we need the expert code writers employed by the WMF to write the tools the volunteers have identified as being needed now and keep asking for, rather than speculate on the positive effect AI or ORES will certainly have, but only in the more distant future.

To expect the volunteer community to write their own code as well would be asking a bit too much - en.Wiki is as big as all the other WMF encyclopedias rolled together and thus demands some extra attention. What the volunteers might well do however, is find their own solutions which the Foundation may disapprove of, but which the volunteers are fully entitled to roll out by creating their own governance structures by which they maintain quality’’.

Rescuing or engaging with new content can be a reward in and of itself, yes, but only if the page reviewers can see potential in the articles they police rather than a disheartening flood of utter rubbish that needs constant mopping up. In the older days of Wikipedia, NPP was interesting, new articles arrived and it was a pleasure to read them, cross a few Ts, and dot a few Is and approve them for inclusion. This is absolutely not happening today. The good stuff is now largely (fortunately) autopatrolled. What we are basically doing today at NPP is shovelling s*** and standing up to our waists in it. And rather than being rewarded, we’re sometimes being criticised for doing it as best as some of us can. Kudpung กุดผึ้ง (talk) 17:45, 5 June 2017 (UTC)

• Re «keep articles from ever being reviewed», this is incorrect. The articles will be reviewed (and edited) across their life. It can happen that an article never gets reviewed only if it's never ever read (there are some articles which receive almost zero page views, most notably certain hoaxes about names which nobody will look for and which have few or zero incoming links). --Nemo 13:46, 16 June 2017 (UTC)

I am one of those editors skipping articles while patrolling

I think this conversation is great, but I also find that a lot of the comments here on the talk page are not approaching one of the biggest conclusions from the WMF team's analysis (note, I work for the WMF, but in a completely different focus/part of the organization): that, at least in their theory, alot of the problem probably lies in "Time-Consuming Judgment Calls".

I am one of those patrollers who skips pages because of the "Time-Consuming Judgment Calls". I frequently will start at the back of the NPP patrol log, and about 30-50 pages in, and about 5 patrolled items later, my experienced editor brain is a bit exhausted with the backlog because:

• I assume most of the items that I am skipping would be low-reward investments for the project -- if I had some kind of filter of importance (pageviews since creation, relative connectedness to other articles), I would take the time to make those judgement calls.
• the items that I am skipping are typically in fields that I just don't care about (schools, sports biographies, businesses) as opposed to ones I wouldn't mind trying to wrap my head around (history articles, towns, albums), or would prefer to work on (academic concepts, literature or film, historical biographies) so I am not going to take the time trying to figure out notability, or quality of sources for flagging.
• I assume that the contributors of that content is going to be too much work to mentor (because they appear to have thrown something beyond my ability to mentor at the time).

If I could do any of the following actions with the queue within the NPP tool, my time looking at the article would be more rewarding:

• flag the article with a WikiProject (like with the Rater gadget), that then could be used to created new filters on the patrol backlog -- that way I could know that there is a good chance of a patroller with domain interest to look at the page. (for example, I use Bambot to do this with Category-based backlogs)
• filter the content by a score that arbitrarily estimates public interest (pageviews x # of editors?) that way I would know that I would be spending time on the complicated judgement calls that count (I am an eventualist when it comes to backlogs: it doesn't really scare me that the backlog is massive, as long as the higher priority bits get taken care of first). I do article assessment for WikiProject Novels, and use the score filter, to prioritize which ones to assess of the multi-thousand article backlogs. This gives me the sense of hacking away at the relatively important, even if the backlog is never ending (for example, I would use this queue to pick the stubs that I would assess for relative importance to novels).
• Filter the content by "likely good faith contributors" or "likely bad faith" via ORES -- so that I could spend time on articles that likely could use some editor-love, for relatively good editors.

More rewarded, I would be more motivated to patrol more pages. Also, I think these filtering strategies, would create very simple ways to engage more experienced editors in the backlog: either via "we have a lot of popular pages, which we aren't sure if they are any good: lets make sure that they don't put Wikipedia in a bad light" or "There is a huge backlog of Rugby biographies that need to be reviewed! You edit Rugby Biographies: come help!" or "There are a lot of Good faith people out there that need help! Help us grow the community by interacting with them!" Also, the content that would get neglected with these filters, would be low public-interest, low topical relevance, probably neutral contributions (i.e. no risk to our long term public impact).

I think what we have here is a big number, that only looks intimidating and/or bad, because we can't prioritize within that number materials/content that deserves more time to make judgement calls. Even if the backlog grows indefinitely (like many of our other backlogs do), with more filtering folks could hack at the bits that make the most sense to them (I disagree with the proposed auto-expire of the backlog-- backlog ≠ bad thing) . Right now the only reasonably good filters (time, no categories, and deletion) are only good for someone motivated by those particular Wikipedia-wide concerns with the content -- which most of our other tools, programs and activities suggest editors aren't. Sadads (talk) 20:49, 2 June 2017 (UTC)

This is an extremely valuable comment Sadads. Your statement: 'More rewarded, I would be more motivated to patrol more pages' is absolutely key to the current issue which is defying any explanation of the shape of the actual backlog growth curve. I will add your bullet points to the community's wish list at Wikipedia:Page Curation/Suggested improvements which I believe at least Kaldari still has on his watchlist. Kudpung กุดผึ้ง (talk) 20:53, 3 June 2017 (UTC)

A very stupid Facebook-esque suggestion

What if we had a bot that recorded the top reviewer for each week, and gave them a barnstar for it? This is stupid as hell, but if we want people to actually push buttons, everything tells us that these stupid pseudo "rewards" help with that kind of thing, and maybe could boost morale overall, since the only feedback most reviewers probably get is negative feedback when someone notices they've been doing it wrong. TimothyJosephWood 20:20, 3 June 2017 (UTC)

Actually (since I'm basically currently bedridden and numb with medication), what if we had an entire designed positive feedback structure in place? You just made your first review after six months? Here's an encouraging comment on your talk page. Thinking of the fr.wiki welcome bot, but it doesn't need to even be attributed to a bot, it could be done by a bot and attributed to one of the coordinators, or just a random volunteer who opts in to be attributed to. TimothyJosephWood 20:23, 3 June 2017 (UTC)

@Timothyjosephwood: I think this actually might work. Many users seem to be fond of/crazy for barnstars. But here is one downside too, we might lose the qulity of reviewing. I am out of my zolpidem. —usernamekiran(talk) 20:42, 3 June 2017 (UTC)

This is absolutely not crazy at all, Timothyjosephwood. The system exists but the promised Barnstars (which I spent several hours designing) at Wikipedia:New pages patrol/Coordination were never issued by the NPR coordinators. Some of us might not be so interested in bling, but judging by the offer of trophies for good work on their drives, other sub-projects, such as for example the WP:GOCE, seem to thrive on the encouragement their rewards bring about. According to soe new data just released, it appears that of the 400+ Reviewers, the vast majority of the reviewing is only being done by around 20 or so of them. Kudpung กุดผึ้ง (talk) 20:57, 3 June 2017 (UTC)

Well, I wrote a 200 word comment but got edit conflicts for days. I've actually awarded this barnstar at least once or twice, once the stats started getting posted on the relevant talk page. It's not really about bling so much as it is about positive reinforcement. As I've pointed out myself in various forums, NPP is a maintenance task, and maintenance tasks are janitorial and not so rewarding. Basically, no one says anything until you fuck up. Mobile apps manipulate this kind of thing all the time, I see no reason why we can't manipulate it to encourage people to do good things they're already motivated to do. TimothyJosephWood 21:05, 3 June 2017 (UTC)

I agree with Timothy.
Kudpung sir, I think the stats are either incorrect, or we are reading them incorrectly. According to this, I have reviewed at least 100 articles. But the stats dont show it. —usernamekiran(talk) 21:14, 3 June 2017 (UTC)

Wikipedia:WikiProject Military history is also a good example of someone using "bling" to recognize top contributors and encourage others. There's a reason why every military in the world uses these kinds of tokens, and that's because when I got my first AAM it felt great, even though I knew it was a low level award, I knew it was something I could wear for the rest of my career and build on. It could be even better if we could adapt an Oak leaf cluster type system, and try to recognize our most consistent contributors. WP:MMORPG aside, people like being appreciated, and this is a way we can do that with almost zero cost to the project in implementing. TimothyJosephWood 21:30, 3 June 2017 (UTC)

Timothyjosephwood Look, I'm absolutely not opposed to the use of bling (except where younger newbies just keep swapping barnstars), it's just that personally, at my age, I'm not bothered much about receiving any more rewards for anything I do in life. I always very much appreciated the barnstars I was given during my earlier editor and admin days - in fact there were, and still are so many of them that I stopped adding them to my barnstar user-sub-page in 2013, but of course I still enjoy getting them. Barnstars are probably hugely motivating for any Wikipedians under 70 ;)

Usernamekiran yiu were only given the right four days ago so you probably don't figure yet in the latest database query.Also, it only lists completed reviews, not the pages that were tagged but not clicked as 'Reviewed/patrolled'. You've actualy done 142 patrols in the last couple of days - keep up the good work! Kudpung กุดผึ้ง (talk) 21:43, 3 June 2017 (UTC)

Well... something like this, if it happened regularly wouldn't just be a reward system, it would also be a type of backdoor advertising (for lack of a better term), to remind fairly dormant talk page stalkers that NPP is a thing that exists, and that other's are actively putting work into. I don't think we need reminding that most of the people who have the right are not actively using it. TimothyJosephWood 07:53, 4 June 2017 (UTC)

After reading the comment(s) above, I completely support Timothy's suggestion. I hope you are feeling better today. :-) —usernamekiran(talk) 11:10, 4 June 2017 (UTC)

A word of warning: The experience at AFC of handing out bling during backlog drives has been a nett negative, in the race for accolades the quality of reviewing declined drastically. The AFC project consequently gave up in backlog drives because cleaning up the messes made by over-eager (mostly newbie) reviewers took a lot of time by experienced less bling-influenced reviewers. It also confused article authors who suddenly found that the review they received so promptly had been revoked and replaced by (usually a more negative) new review. Roger (Dodger67) (talk) 08:53, 5 June 2017 (UTC)

I would expect that to be mitigated somewhat by the fact that this is a fairly thoroughly vetted user right, at least compared to AfC (at least currently). TimothyJosephWood 14:10, 5 June 2017 (UTC)

Really? I'm under the impression that it is the exact same user right for both review systems? Roger (Dodger67) (talk) 19:33, 5 June 2017 (UTC)

That was suggested... I don't remember where the actual RfC was. But AFAIK it didn't gain broad consensus. AfC is still more of an "opt-in and don't get involuntarily opted out" kindof thing. TimothyJosephWood 19:37, 5 June 2017 (UTC)

Rob started the RfC. It didn't get that much participation to be honest (despite being on CENT). I think some sort of merging of the two systems could likely get consensus as a part of a broader packages of reforms and is necessary to moving forward with a better designed system for new content. TonyBallioni (talk) 19:44, 5 June 2017 (UTC)

Thoughts on the relevance of page views

This report is great. It provides perspective on a complex subject.

On of the pieces of information that I found very informative is the number of page views for some example articles mentioned in the report. Page view numbers help to understand how many people may be affected by potential issues in an article, and at the same time, how much demand there is for such content. I was wondering if those numbers can be used to help reviewers in some way. For example, using them to prioritise the backlogs or just surfacing them in reviewing tools to help reviewers in their assessments. This may be an area worth exploring as work is done in the area of review tools.

-- Pginer-WMF (talk) 08:06, 5 June 2017 (UTC)

@Pginer-WMF: I would recommend more of a score of some sort (per my comment above). The best example of something like this which accounts for more features to the articles than what is being measured, is the score at: the 1.0 assessment tool. For example, for NPP, I could imagine us wanting to measure (from heaviest weight to least weight):

• Average daily Pageviews after the item has been in the queue 3-5 days (to allow for the burst of pageviews during the first couple days of a new article and allow for speedy deletion, and other early detection by the folks that work the front of the queue)
• # of links (or maybe even string matches? -- this might be computing intensive) to the page on mainspace pages (to indicate relative demand)
• # of editors beyond the article creator (to indicate relative "preservability" or contributability).
• # of maintenance templates or WikiProjects already on the page (an indication that someone has made some judgement call on the article, but hasn't done the full review of the article for the NPP)

If a score was calculated like this for each article in the queue every 2-5 days (to reduce the number of queries that you would be running), after the grace period for pageviews, you could get some pretty accurate queues of "most in-demand or potentially interesting" articles in the queue, and the stuff that folks have made partial judgement calls on, could be slowly given momentum to be pulled to the top of the queue under this filter. Sadads (talk) 13:54, 5 June 2017 (UTC)

This in particular also comes to mind. TimothyJosephWood 14:10, 5 June 2017 (UTC)

Reviewer participation stats

I added a section to the report with stats on reviewer participation -- from Jan 2015 to May 2017, we've got the number of unique reviewers working each month, and the number of reviewed pages. There are some noticeable changes at June 2016 and November 2016. In June, User:SwisterTwister slowed down their participation, which brought the number of reviews down. In November, with the creation of the patroller user right, the number of active reviewers dropped from around 950 per month to 350 per month. I'd be interested to know what you all think about the newly-added stats. DannyH (WMF) (talk) 20:53, 5 June 2017 (UTC)

Things we can agree on

Thanks for the report. The key points that I think the reports tries to demonstrate are:

• Only a small percentage of the backlog was created by non-autoconfirmed users so implementing WP:ACTRIAL is not a solution to the backlog
• Changes to NPP policy have a role in the increasing size of the backlog
• The current NPP system is not sustainable and WMF will not support a new system that is not sustainable
• The 30-day expiration should be reinstated to make the system sustainable

Other interesting points of discussion here:

• The definition of non-autoconfirmed user used to create the graph potentially underreports the size of the backlog from these users. A revision to fix this is being worked on.
• While there are some holdouts who believe more manpower is a possible solution, there appears to be consensus that the current system is not sustainable
• It does not appear that a consensus to reinstate the 30-day expiration is in reach at this time

Here are some of my own opinions and reaction to the report:

• There's a whole deletionist-inclusionist thing on WP. Deletionist sentiment has been on the rise as the project matures.
• The NPP policy changes that have contributed to the backlog are a manifestation of prevailing deletionist sentiment. It will be difficult to get consensus to reverse any of these changes.
• The WMF rejection of ACTRIAL and this proposal to reinstate 30-day expiration fly in the face of prevailing sentiment
• There is increasing tension between the wish to have the English Wikipedia governed by the editors who work on it and the "Anyone can edit" mission
• Everyone seems to assume that a backlog is a problem that needs to be solved. In some cases, a backlog can be both problem and the solution.

I don't believe Wikipedia editors are going to change their convictions about the project based on decree from WMF. In the current environment, we're not going to get consensus on inclusionist-leaning policy changes and imposing such changes will further polarize the community. New NPP reviewers are not going to come charging over the hill. ACTRIAL has demonstrated that we can get consensus on quality-focused policy. WMF is unlikely to abandon the "Anyone can edit" mission and the deletionist sentiment is not strong enough to challenge this.

It seems to me that the way forward is for everyone to get comfortable with a large and growing backlog. The WMF and inclusionists can console ourselves with the knowledge that backlogged pages lose their NOINDEX status after 90-days. ~Kvng (talk) 21:16, 5 June 2017 (UTC)

@Kvng:

• The current system is not sustainable because more than 70% NPR editors have gone inactive. Getting active reviewers would create a sustainable system.
• There is a 90 days period for an article to be indexed. There are many unencyclopaedic articles out there in the backlog, a few hoaxes, and a lot of promotional articles.
• By closing the eyes, one can't see the problematic object, but closing the eyes doesn't erase that object from existence. The object must be worked on. —usernamekiran(talk) 23:02, 5 June 2017 (UTC)

I acknowledge that there are people who believe more reviewers is the solution or part of the solution. However, the WMF is clearly not buying it and there does seem to now be a consensus of WP editors that agree with them on this point at least.

There is indexed crap in the encyclopedia. The WMF proposal asserts that crap won't get improved until others can find it. Others believe most crap will always be crap. This is just the inclusionist-deletionist tension restated. This tension is not going to go away. Accepting the tension, working incrementally and accepting that there's not a solution that everyone can agree on does not mean burying our heads. ~Kvng (talk) 17:42, 6 June 2017 (UTC)

Usernamekiran, I added some stats on Reviewer participation to the report, and it shows a sharp decline in the number of active reviewers in November 2016, when the user right was created. It goes from around ~930 active reviewers doing 29,000 reviews/month to ~350 active reviewers doing 22,000 reviews/month. The average number of reviews/person goes up, which may mean that the most productive reviewers have survived the transition, but it's a lot fewer people doing fewer reviews.

That's an example of the kind of changes that I think are making the process unsustainable. The standards for an acceptably reviewed page have gone up, the restrictions on who can review have gotten tighter, and the instructions say that people should go slowly and leave difficult reviews to other people. You can't do all of those things, and then expect more people to show up and do more reviews. DannyH (WMF) (talk) 17:50, 6 June 2017 (UTC)

(edit conflict) @Kvng: Yup. I agree with you on that. And the "expiration period" doesnt matter for this issue. The crap that needs to be reviewed is already in mainspace. Any wikipedia editor can see it - can find it. —usernamekiran(talk) 17:53, 6 June 2017 (UTC)

@DannyH (WMF): yes. Its true that screening process has gone very tight for reviewers. And the guidline stating people should go slowly and leave difficult reviews to other people. creates a sort of paradox. —usernamekiran(talk) 17:58, 6 June 2017 (UTC)

@DannyH (WMF) and Kvng: Just found out though Boleyn. Its not a lot, but its a good sign. Progress is being made. Soon, we can decrease the backlog markedly. —usernamekiran(talk) 18:32, 6 June 2017 (UTC)

Autopatrolled

So... how does "the graph" look when you take into account articles created by autopatrolled users prior to their gaining the autopatrolled right? There's been a fairly big push lately to culling the herd in this respect, and fairly mass examining, sorting, and funneling those potentially eligible to PERM, at least it seems that way. I don't actually have statistics on whether there's been an overall marked increase in those having the right as measured by the number of new articles created.

What kind of difference would it make in the backlog if we automatically and retroactively reviewed unreviewed articles by editors when granting autopatrolled? I mean, some of these are by design some of the most prolific article creators on the project. Admittedly, this may effectively raise the bar at PERM, since the responsibility for checking off on potentially scores of articles falls to the button pusher. TimothyJosephWood 21:30, 5 June 2017 (UTC)

Moving forward

I've started a thread at Wikipedia_talk:The_future_of_NPP_and_AfC#Moving_forward to discuss how we can practically move forward with any potential reforms to the NPP process because I feel that might be a better location to discuss the broader questions than on this reports talk page. Comments from anyone who is interested in the discussion would be appreciated. TonyBallioni (talk) 00:46, 7 June 2017 (UTC)

A modest proposal to clear the patrol backlog

After reading the report and a lot of the responses, I have a modest proposal that seems to have been hinted at but not flushed out. Why not allow quicker triage - after you read the article, and don't mark it patrolled, move it into one of the sub-buckets described in the report - in a mobile and desktop friendly way.

1. Category 1: They're probably notable but badly written (like Earl Edgar)
2. Category 2: They're well-written but have questionable notability (like the Luxembourgian women’s handball division)
3. Category 3: They're a weird mix of both (because life is complicated).

These three categories can be called "Patrol-pending articles", versus just "unpatrolled". This has the immediate benefit of letting others know that an article has been viewed at least once, and is not an easy article to patrol.

Category 1 goes to a "copy edit required" bucket, for those who like to copy edit but aren't subject matter experts. That gets marked as something like "viewed but not edited" and is no longer part of the new article backlog. Category 2 goes to a "subject matter expert needed" bucket, with corresponding topic categorization, and the talk page is automatically created with a note that experts in that category are being sought. People could sign up for patrolling rights and specify a category they'd like automatic notification on. I am part of the feedback request service, and while it's interesting getting random articles to help with, I'd be better put to use in technology articles. Patrol approved editors can still review the categorized backlog as needed. This would improve engagement. Category 3 is the hardest bucket - those ones require the most time and effort, so the gamification efforts (barnstars, etc.) may work here - the elite of the elite.

Again, with this process, you're crafting the patrolling challenge to match people's unique skill sets and interests, with a goal of fostering engagement.

Another related issue that I haven't seen discussed is articles for deletion. The afd process is related to this, in that it is another barrier to good information appearing on the encyclopedia. I don't need to name names because many of you will already know people like this, but with the higher standards being applied, and the speed at which some people work, many good articles are being deleted. I've personally seen a bias against articles on Indian subjects, likely because of Western unfamiliarity with Indian culture coupled with the limited English of the writers. I have participated in several afd discussions started by the same people and successfully voted to keep the articles, but can only wonder how many good articles have been lost. OK - so here's another not-so-modest modest proposal - I lied. I recommend that we track nominations for deletion, and the success rates the nominators have. If an article is voted keep or no consensus (against the nominee's wishes), that's scored a -1, while a successful nomination (deletion consensus) is a 1. Once someone has a negative total score (50% of their nominations are successfully blocked) they would be capped as to how many articles they could nominate, per some period. I don't have system statistics but I'm sure the sysadmins can figure out what would work. This would minimize disruption, and preserve the goal of keeping good info on the site. Having to be right 1/2 of the time seems fair - otherwise count all the time volunteers spend protecting info from unnecessary challenges as time which could be better spend patrolling new articles, and solving the first problem. Timtempleton (talk) 17:26, 7 June 2017 (UTC)

There is already a tool for scoring AfD participants. Closing administrators sometimes use that to assess quality of the consensus. (And participants use it to try and undermine each other's credibility.)

Unless I misunderstand it, I don't believe your modest proposal would significantly improves patrolling efficiency. It starts by adding another step to the process. Even if net efficiency is gained by matching people's unique skill sets and interests, it is not likely to be enough improvement to convince the WMF or a consensus of editors that, with this proposal implemented, NPP would be sustainable. ~Kvng (talk) 18:08, 7 June 2017 (UTC)

The buckets solution will be sort of one more step in the bureaucracy. —usernamekiran(talk) 18:25, 7 June 2017 (UTC)

Thanks for your feedback and for showing me the afdstats tool. Since deletion activity is already tracked, it wouldn't be too hard to automatically use it to filter out disruptive nominators. The sysadmins can look into editors nominating articles for deletion that have the lowest correlation with consensus and find ways to correct this, perhaps by politely messaging the person, and if that doesn't work, limiting their deletion nomination capabilities. On the other hand, thinking this through, I suppose that if I was willy-nilly nominating articles for deletion and constantly failing, I'd have other editors monitoring my edits to outvote me, and they could then message me on my talk page and ask me to stop. Further disruption could then lead to my being reported. But as to my first proposal, I don't see triage as bureaucracy - the point of this report and analysis is that something has to be done to clean up new article patrol. Is it better for a troubled article to simply age off the patrol list and get put into the general population, so to speak, due to patrolling difficulty, or would it be preferable to triage it as I propose above, and allow us to faster clear the backlog? Timtempleton (talk) 18:49, 8 June 2017 (UTC)

It's VE's fault.

We need to rid ourselves of the delusion that writing a new article from scratch is easy, and we need to stop telling that lie to newbies. The truth is that it's hard, even veteran Wikipedians don't simply whack out a perfectly formed complete article in one sitting. From my not inconsiderable experience at AFC I've come to the conclusion that the vast majority of wannabe beginner editors are in fact barely functionally literate, even those whose mother tongue is allegedly English.

I suspect that Visual Editor is to blame for the origin of the massive growth in various backlogs. Before VE the requirement to learn at least the elementary basics of wiki-markup, and the consequent necessity to read and understand various guidelines before one could actually write anything, acted as a barrier to entry that fairly effectively kept out the incompetent. Then VE came along and made it dead easy for anyone to post any crap they felt like without first having to learn anything about Wikipedia. VE is malware, plain and simple. Roger (Dodger67) (talk) 19:03, 7 June 2017 (UTC)

I've never used visual editor, but I'm sure it wouldn't be hard to analyze the patrol backlog and see which tools are creating the articles that are most responsible for the backlog. I just worry that this essentially limits contribution rights to those who can code, which would have a dampening effect on new contributions. Timtempleton (talk) 18:51, 8 June 2017 (UTC)

speaking as someone perfect that home with wikitext, and very much aware of the frequent errors still being produced by the visual editor--such as inserting new text next to existing words at the beginning of the sentence rather than where they belong, or the impossibility of editing manually written references, I don't think wikitext is much of a burden. But some people do, and they should be allowed to create articles however they please,

It is totally contradictory to the basic idea behind "everybody can edit" to deliberately set technical impediments for editing. DGG ( talk ) 16:55, 9 June 2017 (UTC)

Maybe we shouldn't set techical limits, but we should deliberately set experience limits. After all, everybody cannot edit the main page or templates.- MrX 20:17, 9 June 2017 (UTC)

This whole discussion has been very educational for me. I'm really surprised to be learning these perspectives from people I have respected in the community and who welcomed me as a newcomer. Is it the mission to stem the flow of free knowledge? Yes, that was a leading question. Here's another: isn't the suggestion of abolishing visual editor in the above context and suggesting because contributors do not know wiki markup, they cannot be valuable new contributors, a bit elitist? Wikipedia content and contributors are representative of a small fraction of society. Let me clarify: it is largely white and male. This is for nuanced reasons, stemming from societal histories.

Anecdotally, when making my first edit with an account, I was sweating like mad. There is so much pressure to be perfect by some members of the community, I was waiting for the backlash. To some point, I still am. Don't tell me I need a PhD in Wiki too. I'm kidding, but sometimes it can feel like that is the expectation seasoned folks have of newcomers.

I know this is not what the intention of the conversation leading to my post was; we, however, cannot ignore these truths in the community if we truly identify with the mission. I'd love to blame VE for the potential of including more people in the process, and sure, like everything else, it can improve too, but I don't think this is the silver bullet to the backlog. And lore has shown us that not all monsters respond silver bullets. This solution is nuanced just like the issue. Jackiekoerner (talk) 15:04, 10 June 2017 (UTC)

Quite obviously we do not want to deliberately exclude any potential good faith contributors, that is indeed counter to the "anyone can edit" ethos. Before VE the need to learn at least the bare basics of wiki-markup was a de facto barrier to entry that kept out many of the malicious and incompetent. VE has removed that barrier for the sake of allowing many more good faith contributors in, which is a good thing of course, but the side effect of filtering out the incompetent has been lost. We need to replace that unintentional function of wiki-markup with some other filter mechanism - which brings us back to ACTRIAL..... Roger (Dodger67) (talk) 15:47, 10 June 2017 (UTC)

The question remains, does the backlog growth correlate with key VE dates - initial launch of VE, when it became opt-in and when it became the default editor? Roger (Dodger67) (talk) 06:48, 15 June 2017 (UTC)

A Proposal - Limited Page Creation

Having read the above discussion, I have a proposal.

1. Pages are auto-approved from the NPP queue after seven days.
2. An individual editor/IP address can only have one page in the queue at a given time; page creation (outside of User space) is blocked if there is a page they created already in the queue. Ideally, good page creations will be approved quickly (less than 1 hour) and this will only be an impediment to bulk creations.
3. Admins can lift the limit for a limited time window (for themselves or for others) to allow for bulk maintenance that creates many pages.

Power~enwiki (talk) 19:24, 7 June 2017 (UTC)

Power~enwiki Your point 1 is tantamount to abolishing NPP. Given the size of the backlog, simply approving everything over a week old means only a tiny fraction of all new pages will actually be patrolled at all. The only way such a short queue could have any chance of working is if we can pre-emptively stop the majority of incompetent and bad-faith newbies from even creating any new page in the first place. Roger (Dodger67) (talk)

Not to mention that implementing ACTRIAL has the positive benefit of being much less biting than this for the good faith newbies who don't know what is and isn't acceptable for a page on Wikipedia. Improving the page creation process via AfC or another means for new users is a vital part of this conversation and needs to happen in addition to whatever other reforms we have. TonyBallioni (talk) 18:55, 8 June 2017 (UTC)

page creation are not going to be dealt with in one hour. In particular, pages that are plausible but apparently incomplete are usually tagged accordingly and deliberately not reviewed for a few days to give new users a chance to finish them. Second,anything remotely tricky or questionable need some thought. There is a point to NPP immediately to catch the dangerous material, but then it is almost always better to look a little later.When I patrol, I deliberately take different parts of the queue. Third, many articles require some degree of knowledge--I will not review articles on poplar music (because by experience I have found I make ludicrous errors); in the other direction, many other patrollers tend to leave the articles of academics to me or someone else who understand that area. It's necessary to spot things that are in fact notable, but where the article has not yet shown it. They should not be deleted via A7 nit prodded or marked for improvement. This takes both thought, and experience. If experience at NPP has shown anything, its that going to quickly causes bad reviewing The tighter the deadline, the worse the review.

I remain amazed that anyone thinks that setting a time limit will do more than sweep the articles under the carpet, or that anyone thinks that sweeping them under the carpet will improve WP. DGG ( talk ) 16:50, 9 June 2017 (UTC)

Perhaps I mis-understand what NPP is supposed to do. Somehow, pages such as Donald Trump's Russian Investigation Interference are created and sent to AfD within 24 hours; I would hope the New Pages Patrol process would handle this type of article on its own. Power~enwiki (talk) 20:40, 9 June 2017 (UTC)

NPP patrollers use WP:CSD, WP:AFD and WP:PROD to dispose of inappropriate material. Submissions that don't meet criteria for such disposal can be marked as patrolled and removed from the backlog. ~Kvng (talk) 23:23, 9 June 2017 (UTC)

In fact, no NPP except an administrator can delete an article, and almost all administrators, including myself, prefer to list articles for deletion by one of the various processes, and let another admin judge. There are none of us who does not make an occasional mistake, and this double review by two editors is necessary to avoid an unacceptable level of erroneous deletions. The role of NPP is to get pages such as you have mentioned sent to the appropriate process. DGG ( talk ) 23:31, 11 June 2017 (UTC)

the long tail

Something I'd like to bring to everyone's attention is how difficult it is to reduce the backlog as a result of a number of design decisions by the developers of the page curation tool.

The analysis appears to show that reviewers leave the hard cases for last, where they require time-consuming judgment calls. That time would be better spent working on those areas where there is a higher ratio of pages that should be speedily deleted, PROD'ed or sent to AfD. I did not really see anything in the analysis that showed me what reviewers are actually spending their time on. So, as a reviewer, I'd like to provide some insight into how I use my time and how my time can be used more productively by making some changes to the tools and processes that I use.

I can only work at the beginning or the end of a log, and I try to do both; the beginning to catch the really blatant cases that need to be deleted, and at the back of the log because it's interesting and challenging and is a last look to see if we are going to let something sneak by, often those are contributions by paid editors or editors with a COI. The problem is that the back of backlog is so massive that it is impossible to reach. If I look at page curation now, the oldest pages are from 2007, old redirects that haven't spent all that time in the queue. The cases that have been sitting there all the time are from 29 December, 163 days ago. To access a page, using the page curation tool, that is actually at risk of becoming indexed, rewarding the spammers, I would need to scroll to May 12 or thereabout and it would take me about half an hour to do so, if I didn't know to go to Special:NewPages and set the offset manually to 20170512000000 where I can find those pages that are about to get indexed. Pages are only listed at NewPages for 30 days though, so to get at any of the unreviewed pages between 30 and 160 days old, you would have to resort to other tricks.

I can't say what other reviewers have figured out, but I am, for all intents and purposes completely unable to work on the "middle" of the backlog. In my experience, the pages that are 150 days old are no more difficult to review than pages that are 30 days old. They have not been skipped endlessly. They have only been skipped for the first 30 days. After that, they cannot be accessed, and so they are just sitting there until they reappear at the end of the backlog.

Another problem that I face are the varying notability standards that the different projects have developed. I'm pretty familiar with art-related stuff, but know nothing about sports. So I much prefer to review articles that cover a subject that I care about, and where I have access to sources as well as a good sens of which sources are reliable. So I do skip a lot of wrestling articles because I can't tell all the different wresting organizations apart. A lot of my time goes to skipping stuff I don't care about. To work around that problem, I look at suggestions from InceptionBot, which gives me project-related topics that I do care about. I work faster, I'm happier, and I contribute more reviews when I can work in areas where I have expertise.

Another time-sink for me is performing manual tasks that could be automated. Checking what links here to see if something is an orphan when the page curation tool already shows me that an article is an orphan seems unnecessary. I should not have to tag an article as missing references when the page curation tool already has that information. It would be nice to preload the uncategorised, orphan, unreferenced tags if an article is already listed in the New pages feed with No categories, Orphan and No citations.

Checking recreations of deleted articles when dealing with spammers is another manual task that is taking a lot of time. Maybe there is a better way to do this, but bringing up the deletion log is time-consuming. Automated flagging of recreations would be helpful. For checking copyvios, I have added stuff to my toolbar that loads Earwig's tool with the page I'm reviewing to make that a bit faster, but it would be nice if that too had been done already.

In summary: The page curation tool would already serve my needs (and possibly others) much better if only two improvements could be made: a filter for pages that I have already skipped, and a configurable offset date.

Thanks for listening, and thanks for all your hard work Mduvekot (talk) 23:11, 10 June 2017 (UTC)

If we categorized new articles even roughly, we would be much more easily able to patrol where we had some knowledge. A very simply approximate keyword approach would do, and would be very easy to implement. It might even be so easy as not to need the WMF). (I have been asking for this for many years now). DGG ( talk ) 23:34, 11 June 2017 (UTC)

I agree with and appreciate Mduvekot's post. Like DGG, I would really like a keyword search, I could be much more effective. A third opton between oldest and newest would be great too. I'd rather prioritise those not brand new and still being worked on and not yet indexed. Boleyn (talk) 05:25, 12 June 2017 (UTC)

I agree with Mduvekot's thoughts here, and will again emphasize, there needs to be significantly more automation built into the review process. The system should triage new articles into categories based on degree of likelihood of deletion, and in some case should take other actions that human reviewers would take, for example:

• Move unreferenced articles more that 10 days old to draft space.
• Check for copyright violations and mark them for deletion.
• Find G4 candidates and nominate for deletion.
• Triage articles into categories (low, medium, high risk of being undesirable). This could be done by evaluating the number of editors of an article, number of other articles edited by the article creator, article title appearance in Google search, etc.
• Many more such rule sets could be defined to automate a significant portion of the the NPP queue.

I know it's controversial, but I would also support automatically marking articles as reviewed based on time since created and the extend to which other editors have already reviewed an article. For example,

If (article age=30d & number of extended confirmed editors >=3 & number of references [including non-social media external links] >=2 & (article not an obvious copyvio | G4 candidate) & article has at least one tag) then mark as reviewed

- MrX 12:27, 12 June 2017 (UTC)

Mduvekot, thanks for posting your thoughts. The developers on the page curation tool didn't include searching by date because, at the time they worked on it, NPP was thought of as a triage system. People were encouraged to look at a page, make a decision, and move it out of the system. In the system they were building for, people didn't have to go back three months to find pages, because those pages had already been checked and cleared.

There are a lot of good ideas for feature improvements, on this page and on the Future of NPP page. If we want to make progress on those, we have to have some discussions about what NPP is for, and what a workable system ought to look like. Would we want to build a system that's optimized for a five-month backlog? DannyH (WMF) (talk) 23:36, 12 June 2017 (UTC)

There are a lot of good ideas too at Wikipedia:Page Curation/Suggested improvements, most of which would contribute to making the excellent suite of curation tools the first choice for qualified patrollers. They just need prioritising rather than listing at Phab and being ignored there.

We certainly don't want to build a system for a five-month backlog. It's also important to remember (as stated many times already) that the characteristics of articles being created today are not the same as when WereSpielChequers, The Blade of the Northern Lights, and I began discussing the backlog 7 years ago when incidentally it was possible for one of us to clear the backlog in a 3-hour session), and consequently with the build of Cuartion.

With over 400 qualified reviewers the backlog should only be a few days, but it appears that around 90% of the reviewing is being done by only 10% (or less) of the people who asked for the reviewer right - and some of the more prolific reviewers in that 10% are admins who have the right anyway; I therefore request once more a proper examination of the shape of my chart, and why the excellent graphs here cannot be reproduced to reflect current trends over, say, the last 12 months. To say that since the right was introduced we now have only 300+ reviewers instead of around 900 is deceptive - what we have done is successfully prevented 600 new and/or inexperienced editors wrongly tagging articles for deletion and letting toxic articles and spam go through.

Keyword filters may be useful for experience users who which to apply their subject knowledge, but it will only accelerate the picking of low hanging fruit. We need objective solutions, not palliatives such as hiding the backlog from sight. Kudpung กุดผึ้ง (talk) 01:21, 13 June 2017 (UTC)

In response to DannyH (WMF). I don't think anyone wants to build a system that is designed for a 5 month backlog. My argument wasn't that we should design such a system, but that if you have a triage process that only allows that cases be processed at one point or two points, the stuff that's in the funnel now will sit there for the next five months, and will remain there until it comes back out at the other end because nobody can look at it. You seem to suggest that if only we went back to doing triage, we could clear the backlog. My understanding of triage is that it should look something like this (and isn't this what we're doing now?)

articles that are fine as they are -> keep

articles that cannot be saved, no matter what -> delete

articles that can be improved with editing -> tag

Perhaps the triage criteria should be changed, so that you would have this:

articles that are fine as they are or can be improved with editing -> keep

article that must be improved before they can be accepted -> move to draft

articles that cannot be saved, no matter what -> delete

We need to have a conversation about adding move to draft to the reviewer toolbox. Mduvekot (talk) 12:23, 13 June 2017 (UTC)

Tracked in Phabricator
Task T124396

What we are doing (or should be doing), Mduvekot, ever since we got the Draft namespace created (and that's why we wanted it) is:

• articles that are fine as they are -> keep
• articles that can be improved with editing -> tag and keep
• article that must be improved before they can be accepted -> move to draft
• articles that cannot be saved, no matter what -> delete

and we've been having the conversation about the 'move to draft' tool since first suggested here 13 December 2015 by czar and listed at Phab by Czar on Jan 22 2016,, but despite being requested several times, and listed here 9 months ago, the devs are not doing anything about it. Kudpung กุดผึ้ง (talk) 15:45, 13 June 2017 (UTC)

I asked in several venues about adding "move to draft" to the page curation tool but never got far. Instead Evad37 did me a favor and made a script that works really well: User:Evad37/MoveToDraft. Highly recommended  czar 15:54, 13 June 2017 (UTC)

Yes, czar, an excellent little tool and probably put together in just a few hours, but the point I'm making is that we shouldn't have to spend our free time making up for the shortcomings of those specialist IT departments who are paid to maintain core Wiki extensions - which we don't have access to. Kudpung กุดผึ้ง (talk) 16:48, 13 June 2017 (UTC)

You're right. This could be incorporated into Twinkle or written as a stand alone script pretty easily. I have user notification that could be used in some form.- MrX 12:18, 15 June 2017 (UTC)

Peer review

I saw this in the Economist and thought it might be interesting to Wikipedia leadership. It's a discussion of how to encourage peer review, tangentially related to clearing the NPP backlog.[[2]]. Timtempleton (talk) 14:41, 12 June 2017 (UTC)

Interesting, but yes, very tangentially. What a lot of people don't realise is how hard it is to motivate unpaid volunteers to do the less appealing aspects of what they signed up for. It's a bit like herding cats, really. Kudpung กุดผึ้ง (talk) 02:23, 13 June 2017 (UTC)

Some stats

I did some digging around after coming across this question on Quora. I came to a conclusion that only about 16.7% articles on Wikipedia have been marked as reviewed. Please see the calculations here. For those too lazy, there are about 5,422,965 articles on Wikipedia and only about 905695 have been marked as reviewed. Don't know if it helps but thought it wise to share it here as well. Yashovardhan (talk) 19:14, 12 June 2017 (UTC)

• Copied this from WT:NPP/R Yashovardhan (talk) 05:55, 13 June 2017 (UTC)

References

I'm not sure about this, but don't forget that Curation was only introduced in 2012, and that it only logs patrols done through it and not through Twinkle. Kudpung กุดผึ้ง (talk) 07:31, 13 June 2017 (UTC)

Ofcourse, I know that. I just found it interesting and worth sharing. Yashovardhan (talk) 07:59, 13 June 2017 (UTC)

Analysis of the analysis and questions for WMF

1. In the section 'Non-autoconfirmed contributors', the author writes "According to our calculations, there's an average of 1,180 new mainspace articles created on English Wikipedia every day,...",. The graph shows page review backlog, and for only six months. The growth of the backlog is merely the symptom of potentially many other interrelated factors. Would the WMF please provide graphs/query data that include the past 18 months of data, showing the following?

• Number of new articles, by day, created by users who were not autoconfirmed at the time the article was created
• Number of new articles, by day, created by users who were not autoconfirmed at the time the article was created, that have been moved to draft or userspace
• Number of since-deleted new articles, by day, created by users who were not autoconfirmed at the time the article was created
• Number of new articles, by day, created by users who were not autoconfirmed at the time the article was created that, to date, have not been marked as reviewed
• Percent of users who were not autoconfirmed at the time an article was created who have since edited any other articles compared with the total
• Number of reviews, by day
• Number of reviewers who have reviewed each day, by day

This would give us useful insight to new article creation trends, reviewing trends, and article quality trends. It will be far more useful than showing the overall backlog. Thank you for your help.- MrX 12:59, 15 June 2017 (UTC)

2. In the same section, a conclusion is made:

"While removing pages created by non-autoconfirmed users would reduce the burden on that first wave of reviewers, it would result in the loss of many potential good articles. It would also send a clear message to new Wikipedia editors that their contributions aren't wanted, potentially stunting the growth of the editing community. Most importantly, it wouldn't actually solve the problem of the growing New Page Review backlog."

This seems to be built upon several false or dubious assumptions, specifically

• "Pages created by non-autoconfirmed user would be removed" - No, users would be greeted with a message telling them there are several options for creating an article. For example, that they can create a freeform draft, which they can ask an editor to review. If it is ready, the editor will publish it for them.
• "It would result in the loss of many potential good articles" - No, 80% of new article by new contributors are junk. The other 20% would not be lost at all. Evidence is required that there would be a loss of many potential good articles, as that is a rather extraordinary claim.
• "It would also send a clear message to new Wikipedia editors that their contributions aren't wanted" - No, it would send a clear message to new contributors creating the 80% junk articles that we have standards, and to the new contributors creating the 20% usable article that their contributions are welcome but, that they must follow one of the processes before their article can be published. Isn't this exactly what we do for image uploading?
• "It wouldn't actually solve the problem of the growing New Page Review backlog" - Of course it would. It would free page patrollers to review the rest of the backlog and it would disuade spammers and vandals from co-opting the encyclopedia for their non-encyclopedic purposes.

- MrX 22:49, 15 June 2017 (UTC)

Your assumptions seem to be quite worse than those you are challenging. In order.

1. The wiki way works better that obscure Nupedia-like systems which reduce the number of eyeballs on the text to be reviewed. (As shown by the consistent failure of WP:AFC and similar catastrophic attempts.)
2. I don't know where you got those 80/20, but for instance unregistered users create better articles: m:Research:Wikipedia_article_creation#Summary. My explanation: when you block unregistered users from creating articles you lose a good portion of the good ones, while the bad faith users who just want to promote something will certainly register and reach whatever threshold you put. The higher the threshold, the more likely than only bad faith people pass it.
3. Users don't generally get the message you expect. In the best case they accept that there are good intentions behind certain rules, but keep questioning them. More commonly, they just don't understand how a certain process can work for good and assume that it was intentionally designed to work "bad" (for them or what they care about).
4. The worst spammers and vandals are certainly not scared by restrictions.

--Nemo 13:39, 16 June 2017 (UTC)

Perhaps, but my assumptions are based on my substantial experience patrolling new articles. My specific responses:

1. Most importantly, the 80% bad articles will not be published, which is a benefit to our credibility as an encyclopedia and the workload of page reviewers. Only a portion of that workload would transfer to AFC if at least some spammers and vandals are repelled.
2. The 80/20 is in the graphs on this page. The six year old study you cite is way out of date and flawed in ways that have been documented by other Wikipedians.
3. That's an opinion that doesn't jibe with my own observations. I have found that people who come here to help build the encyclopedia generally embrace our processes.
4. We're not trying to scare them; we're trying to make it more difficult for them. We're trying to increase the cost (effort), and reduce the benefit (publication).

- MrX 14:12, 16 June 2017 (UTC)

Deletion statistics

New User deletions

Autoconfirmed User deletions

Comparison of article status

For those who have not yet seen it, we have some numbers based on deletions from the WMF. I've produced the charts based on them. This is based on taking the articles created the first week of November 2016 and checking their status as of 14 June 2017. You can see the data at User:MusikAnimal (WMF)/NPP analysis. WMF is working on getting us more numbers on this, but these also give us a snapshot of what has happened to articles that have all been reviewed. TonyBallioni (talk) 13:14, 15 June 2017 (UTC)

Thank you for posting this TonyBallioni. This seems to show that 80% of articles created by new users are deemed not suitable for main space. Does this data also account for articles moved to Draft: and User: space?- MrX 13:56, 15 June 2017 (UTC)

I assume that articles moved to draft or user space are captured under the category 'other'. Perhaps MusikAnimal could confirm this.- MrX 14:02, 15 June 2017 (UTC)

You edit conflicted with me, but beat me to the punch: I was going to ping him since he pulled the data. I just put it into a spreadsheet and told Google to make the graphs. TonyBallioni (talk) 14:05, 15 June 2017 (UTC)

Just here to say there is more comprehensive and detailed data coming today, over a larger period of time, and much more data to follow! Stay tuned :) — MusikAnimal ^talk 18:20, 15 June 2017 (UTC)

MusikAnimal, thank you so much for your work on this! TonyBallioni (talk) 18:47, 15 June 2017 (UTC)

MusikAnimal Yes, just another note to thank you for all your help and work with this. jcc (tea and biscuits) 19:21, 15 June 2017 (UTC)

Yes, thank you MusikAnimal for your work on this.- MrX 20:18, 15 June 2017 (UTC)

Thank you MusikAnimal, and super excited for the bigger dataset. Sadads (talk) 22:14, 15 June 2017 (UTC)

Updated graph and statistics on the backlog by new vs. autoconfirmed article creators

Hi all,

as planned earlier and also suggested above by Innisfree987 and others, we have produced a different version of the backlog snapshot graph. To recap the explanation posted earlier, the version of the graph used in the report was showing the number of still unreviewed articles per day as it appeared to new page patrollers at Special:NewPagesFeed at one point in time (May 25), using the same definition of "new editors" as patrollers see there (i.e. if you change settings there to only show unreviewed articles by "new editors", you would get a list that exactly corresponds to the blue part of the graph). The new graph instead uses the autoconfirmed status at the time of the article's creation, because that's more pertinent to the question of how much disabling article creation for non-autoconfirmed users might reduce the backlog:

This chart shows the number of new pages that were still unreviewed on June 7, 2017, by page creation date, and split by whether the page creator was an autoconfirmed user or not at the time of the page's creation. See also Data source

It confirms the earlier observation that pages by non-autoconfirmed users (by either definition) make up only a small portion of the backlog - to be exact, just 15% of the still unreviewed pages came from users who had not been autoconfirmed when they created the article.

Regards, Tbayer (WMF) (talk) 22:04, 15 June 2017 (UTC)

Oh, that's great -- thank you, Tilman! I replaced the graph on the report with the new version. DannyH (WMF) (talk) 22:13, 15 June 2017 (UTC)

This is great! Thank you!Sadads (talk) 22:14, 15 June 2017 (UTC)

Yes, thank you, and I really mean that! The graph makes much more sense than it used to. It is useful for looking at the backlog as it is now, but, if I am reading the multiple explanations right, is still not that useful for the question of whether or not restricting page creation would be helpful since it only looks at unreviewed pages, and not all pages at the time of creation like MusikAnimal's numbers do (and yes, I'm aware of the WMF not 1:1 tradeoff theory, but even a 6:1 tradeoff would aid in slowly reducing the backlog). TonyBallioni (talk) 22:21, 15 June 2017 (UTC)

This is interesting, but I don't see any useful conclusion conclusion coming from this information. Maybe I'm missing something.- MrX 22:24, 15 June 2017 (UTC)

MrX: Nothing earth-shattering, but it's another piece of information that helps us get a fuller picture. We'll get some more stats coming soon. DannyH (WMF) (talk) 22:32, 15 June 2017 (UTC)

Thank you DannyH (WMF) As a piece of information, it is certainly useful in context with other information. The three charts in the previous section are very valuable. Addition stats will no doubt help paint a fuller picture. Please also see my request two sections up. I'm hopeful that some of those statistics will be forthcoming.- MrX 22:56, 15 June 2017 (UTC)

MrX: Yes, I saw them, thanks. The graph that Tilman's added here answers one of the bullet points on your list: "Number of new articles, by day, created by users who were not autoconfirmed at the time the article was created that, to date, have not been marked as reviewed." That's all the blue bars on this graph. MusikAnimal is currently working on a bunch of graphs that should be up soon -- the only reason why they're not up yet is that he keeps thinking of something else to add. I'm not sure if that'll address every one of your bullet points, but once he posts the new stats, we can take a look. DannyH (WMF) (talk) 23:54, 15 June 2017 (UTC)

Excellent! Thank you.- MrX 00:34, 16 June 2017 (UTC)

• ...It confirms the earlier observation that pages by non-autoconfirmed users (by either definition) make up only a small portion of the backlog - to be exact, just 15% of the still unreviewed pages came from users who had not been autoconfirmed when they created the article.
  Which would seem, if I am looking at this right, that the pages by autoconfirmed users are harder to patroll. Which in turn means that the low-hanging fruit, the pages from new users, are so awful (which they are) that they are probably easier to recognise as sub-standard. What we need next are the graphs showing deletions. What I am ultimately hoping to see are the same kind of graphs as those made by Scottywong in 2011. Kudpung กุดผึ้ง (talk) 02:25, 16 June 2017 (UTC)

• 
• 
• 

• • Or it means that the hard cases are more likely to be created by autoconfirmed editors. Which makes sense, because if you are a bad faith editor you will be careful to register, make a few edits, try to look legit, then publish a text which looks good and doesn't attract attention. Unregistered users for instance create better articles (see m:Research:Wikipedia_article_creation#Summary), probably because a larger portion of them is in good faith (they didn't even bother to register in order to deceive better). --Nemo 13:25, 16 June 2017 (UTC)

Deletion and backlog statistics - Round 2

Hello! After a few days of coding I think I have a months worth of rich data that I hope will answer some questions. There are a LOT of charts I want to share, so instead of further clogging up this page I'm going to link you to User:MusikAnimal (WMF)/NPP analysis. The data you see there right now covers February 15 to March 15 2017, which should (seemingly) line up with the abrupt upward trend on Kudpung's NPP backlog chart.

No matter what you make of the data, I've got some exciting things to tell you. Thanks to the incredible work of Yurik with mw:Extension:Graph, the charts you see will automatically update as new data comes in. They can all be shown using the {{User:MusikAnimal (WMF)/NPPChart}} template (which I will document fully). You can use this template anywhere and the charts will automatically update. Eventually I hope to have many months, even years, of data. When we get to that point the charts will magically become interactive and you can zoom in/out, etc. For now they are static because I wanted to get the numbers to you quickly, so I didn't put too much effort into it :) I also realize the pie charts don't actually show any numbers, something I hope to fix tomorrow, and I'll also throw in percentages.

One of the things people have been complaining about is how hard it is to get this data. Well, since these charts automatically update when new data is supplied, we just need to automate generating that data. I have the script to do it, I just need bot approval and some rough community consensus. Let me know if you think this is a good idea. My thoughts are to first backfill as much data as we can, then have the bot automatically add data in everyday. That way you always have rich and up-to-date numbers at your disposal. I'd also like to move {{User:MusikAnimal (WMF)/NPPChart}} to Template:NPPChart but I first want to see what you all think.

Now, the chart you all probably want to see the most:

...which is produced by {{User:MusikAnimal (WMF)/NPPChart|type=line|usergroup=non-autoconfirmed|metric=survival}}

Maybe this is suggestive that ACTRIAL has some merit. For the record, I'm indifferent and am more committed to just bringing you the data. The only thing I'll be sad about is half of my charts will no longer be relevant because there will be no new pages created by non-autoconfirmed users – but that's not a real argument but rather your common engineer frustration ;) We can still make use of the other charts.

One thing I did want to point out, however, is that while we have data on the types of deletions (speedy vs PROD vs AfD), we don't have data on the types of speedy deletions. I'd like to do that next, and will share a chart for it once I have the data. What I'm getting at is probably much of the speedy deletions from new users are vandalism-related. I'm not outright dismissing ACTRIAL, but I think it's worth noting vandals will be vandals, and if they can't do it through new pages they'll likely _target existing ones. This is opposed to new users who don't understand inclusion criteria. The ratio of those versus vandals is something I'd personally like to see.

Let me know what you think of the charts, and while discussion ensues I'll keep my script running to backfill more data, and they will all magically update. Again let me formally ask for feedback about the idea of a bot. I'd love to make that happen so we don't have to go through this long process again. With the bot, you won't need to request the data, it will just be there, right here on the wiki :)

Warm regards, MusikAnimal (WMF) (talk) 01:54, 16 June 2017 (UTC)

MusikAnimal (WMF), thank you so much for this. I really can't say how much I appreciate getting all the data presented without much commentary. I'm going to take the liberty to post the backlog graph here, because I have questions and it relates to the other graph presented today and paints a slightly different picture, so I think people should see it on the same page.

These numbers seem to suggest that autoconfirmed and non-autoconfirmed article creation is roughly equal at ~350-375 a day on average. Could you let me know if my reading of this graph is correct? Also, could you provide the raw data in tables at some point like you did a few days ago (not needed quickly, just something I'd like to see). Again, thank you so much for this. TonyBallioni (talk) 02:06, 16 June 2017 (UTC)

@TonyBallioni: Raw data is at Special:PrefixIndex/User:MusikAnimal_(WMF)/NPPChart/Sources/. Your reading of the graph is correct, and frankly I'm a little confused, as I was under the impression that non-autoconfirmed users created considerably fewer articles than autoconfirmed. I'm going to put in more data so we can find out how this compares over a larger period of time. Also one thing to note is redirects are not included, and it would make sense that redirects are usually created by autoconfirmed (more experienced) users, so that might have skewed our stats in the past. This set of data is the first time (that I'm aware of) where we have reliably excluded pages that were redirects with their first revision MusikAnimal (WMF) (talk) 16:27, 16 June 2017 (UTC)

• A quibble: the assertion that most new pages by new users that are deleted are deleted for vandalism doesn't match my experience, nor the data I can pull. Manual analysis of the last 24 hours' worth of deletions from articlespace (raw data; I haven't tried to separate by autoconfirmed or not) gives me the following totals:

Extended content
count reason 10 a1 1 a1+g3 7 a10 9 a11 1 a2 12 a3 1 a3+a7+g11 123 a7 1 a7+a11 26 a7+g11 1 a7+g12 1 a7+g2 1 a7+g4 2 a9 38 afd 8 blpprod 5 g1 3 g10 73 g11 7 g11+g12 2 g11+prod 15 g12 7 g2 10 g3 1 g3+g12 9 g4 17 g5 30 g6 16 g7 41 g8 6 iar 69 prod 3 r2 3 r3 1 rfd 1 u5

I'd have expected more from afd and less from prod, and of course the data set is small, but a7 and g11 are way out in the front as I'd expected. —Cryptic 02:35, 16 June 2017 (UTC)

Thanks for putting this together! I'm going to get this data over a time period and be able to break it down by articles created by autoconfirmed vs non-autoconfirmed users. From your dataset alone I'm a little alarmed of how many G5's there are! It will be interesting to see how things look when broken down by user group MusikAnimal (WMF) (talk) 16:28, 16 June 2017 (UTC)

• The page at User:MusikAnimal (WMF)/NPP analysis is excellent - thank you. It's even more than I had hoped for because I will certainly admit to having been very suspicious that the Foundation was not wanting us to have this information. The stats are very exiting and more than enough to prove that of course ACTRIAL needs to take place - stressing once more that as its name implies, it is to be a trial, not an irrevocable feature, ultimately demonstrating that the restriction would not have a significant impact on the creation of acceptable new articles. And that is why I would really like the Foundation to thoroughly look at WP:ACTRIAL to see how well we prepared for it in every aspect. Kudpung กุดผึ้ง (talk) 02:46, 16 June 2017 (UTC)

  I actually completely disagree with this interpretation: if most of the pages can be speedy deleted, and don't require the slightly more intensive AFD/PROD processes AND 1/4 of the pages survive: we are actually very successful at creating new content via folks who have little or no understanding of our workflows. Instead of blocking article creation: those metrics suggest to me that we should be providing better guidance for folks trying to create new pages (such as a required multi-page tutorial for new editors-- with a short primer on referencing and notability). Speedy has relatively low overhead, and is not the hard decisions that we describe elsewhere (which is where our decision making is not working in New Page Patrol). 1/4 of the content that is keepable, still maintains a lot of topics that we would never have, had new users not tried to contribute. I think the question is more about "how do we not bite users with deletion". Sadads (talk) 03:17, 16 June 2017 (UTC)

  Also there are other considerations besides this particular set of issues related to NPP: though I personally think that most editors shouldn't be writing new articles during events like editathons (and Art+Feminism has demonstrated in that last couple years, that folks make better contributions on established articles), many valuable contributions come from users at events being run without our knowledge or the support of admins: so these editors need some forum for creating that potentially highly valuable content in a forum that won't get lost and require multi-day followup (like AFC). Sadads (talk) 03:23, 16 June 2017 (UTC)

Not enough if the pages are being deleted fast enough because the WMF won't make the Curation software attractive enough for patrollers to use it and refusing to create the landing page you mention. If you follow these discussions you will see that of the 400+ users who have asked for the reviewer right, only a dozen or so of them are actually doing any work. Once the new Reviewers have had a taste of what awaits them in the cesspit of NPP,they are so disgusted at the rubbish they find there, they're discouraged from doing it. You should know, you've patrolled enough pages yourself. The Foundation just wants the trash cosmetically swept under the carpet to make their stats look good. Kudpung กุดผึ้ง (talk) 05:35, 16 June 2017 (UTC)

Awesome, I'm really happy to see the new stats up. One question re: ACTRIAL is whether it's worth losing the ~100 good articles a day created by non-autoconfirmed users, if the current system is doing a good job of speedy-deleting the bad ones? Obviously, I know that nominating and deleting the ~250 bad articles is time that could be spent on other things. But there are other ways of saving reviewers' time, for example: moving back towards a page-triage model, where bad pages get tagged and deleted, and mostly-good pages are reviewed quickly and released from the queue.

The point that I'm making in the report is that the NPP system has changed over the last few years from a relatively efficient page-triage system to a relatively inefficient page-improvement system. Blocking people from creating probably-bad articles because you want to spend time improving mostly-good pages means that this really isn't a triage system anymore. DannyH (WMF) (talk) 15:47, 16 June 2017 (UTC)

@DannyH (WMF): You keep saying that we have moved away from a triage model, but where is the evidence for that? All the report says is that the instructions have grown – well, that's what instructions do. I'm not at all convinced that reviewers' behaviours have changed. The primary purpose of NPP, as given in the instructions, remains to "identify articles which do not meet the criteria for inclusion and/or to tag them for any glaring issues that need attention". The rest of the page lists a lot of additional, optional tasks as well, but a large proportion of it is just further elaboration on that primary purpose: how to check for copyvios, COIs etc. It even explicitly discourages NPPers from getting sucked into fixing articles themselves: "New Page Reviewers should not feel obligated to mentor new users or complete their articles". Is this not all describing a triage system? I for one still review that way.

For my part I agree fully with your analysis that "time consuming judgement calls" are the crux of the problem. However, I don't agree that "scope creep" has created that problem. Rather, it is because as Wikipedia as matured and grown in prominence people have come to see Wikipedia as a social media or promotional platform rather than encyclopaedia, and consequently the subject matter of new articles has shifted. We're bombarded by COI-riddled articles on obscure people, products and international pop culture and it's time consuming just figuring out whether they belong in the encyclopaedia. – Joe (talk) 16:25, 16 June 2017 (UTC)

@DannyH (WMF) and Joe Roe: I am not particularly convinced that the process has changed all that much for those of us who do it: I needed the reminder of the backlog to come back to participate not a change in the documentation to return -- its more of a social engagement and organization problem (there are tons of admins who also, by default, have the right: but don't use it). I think there are other things that have been identified by the conversations here:

1. The policies described for NPP, the access to or understanding of the tooling/process, and the addition of the new user right have created a bottleneck of some sort (still not 100% clear) in the number of people who feel empowered to work on the backlog en-mass. Possible solution: The documentation may be a bit more daunting than it needs to be and the right requires users to "get" that there is a queue and ask to be involved in it: which means we ought to do some more revision, recruiting and engagement one way or the other.
2. Neither ACTRIAL nor a change in process that still provided sufficient triage, would make time-consuming decisions less time-consuming for folks who sincerely want to make sure that marginal articles are given a best chance. I think this is where there is at least some consensus around need for change among WMF folks working on this AND volunteers. Possible solution There are a number of tangible solutions on this page and the wishlist for the NewPagesFeed tool, which could be relatively low hanging product features. However, it doesn't make sense to keep dis-paring that WMF hasn't invested in the tool in the past, when the whole premise of this analysis and the amount of staff time being spent on this, is that they are researching and evaluating which things can be prioritized on their end. WMF clearly cares at this point in time, so it makes sense to look for compromise.
3. We also need to do some 5 Whys on Kudpungs repeated assumption "Once the new Reviewers have had a taste of what awaits them in the cesspit of NPP,they are so disgusted at the rubbish they find there, they're discouraged from doing it": Why #1) Because new contributors create a fair amount of poor content. Why #2) Because they don't make a good faith effort to follow our policies. Why #3) because they have no clue how to follow our policies . Why #4) Because no one told them that we had policies. Why #5) Because page creation's documentation is very limited for new users, and can be easily missed.

Basically what I am hearing in Kudpung's comments, is that we aren't doing a good enough job dissuading folks from contributing the blatantly bad, while also ensuring that the good (at least a quarter of that content and probably some of the content that gets deleted) gets a healthy opportunity. Potential solution:To me, this does not warrant an ACTRIAL, but some more heavy handed measures in the article creation process to get people to slow down and think "is this appropriate for Wikipedia and is it worth my time to go through the steps to do that". For example, we could write a guided tour (Help:Guided tours) or a series of questions (like on the Upload Wizard), which force folks to slow down and if they are good faith, make more of the content either readily acceptable or more clearly out of scope for NPP instead of hard decisions. Some of the bad-faith stuff would be desuaded, much of the good-faith but bad would be better or desuaded, and we would have much less dross in the kilns.

I hope that analysis of what I am hearing helps. Also, I want to note like I did higher on the page: I work for WMF, but have not engaged with or supported this team working on this project during this project as part of my work -- my scope and focus is on WP:GLAM.Sadads (talk) 21:01, 16 June 2017 (UTC)

I feel like what would be a good compromise would be to allow, for a time, the automatic speedy deletion of unpatrolled articles created by non-autoconfirmed users marked for speedy deletion by new page patrollers. This would essentially allow for bad pages created by non-autoconfirmed users to be deleted very quickly, thus reducing the workload. RileyBugz^会話_投稿記録 18:58, 19 June 2017 (UTC)

RileyBugz, you have a better chance of the WMF deciding to implement ACTRIAL on a permanent basis before close of business today than getting that proposal through WT:CSD or any other community RfC process. There's too many complaints already about overzealous CSD tagging by NPP already, and I don't think you're going to find much appetite for this. Not to mention that simply directing users to draft is a whole hell of a lot nicer than deleting things they already published. TonyBallioni (talk) 19:12, 19 June 2017 (UTC)

Is there anything better than ACTRIAL? Or can we do something to give those who aren't autoconfirmed but start out in good faith confirmed status? RileyBugz^会話_投稿記録 19:25, 19 June 2017 (UTC)

• An extremely stupid question: Would somebody please tell how many articles were created, and reviewed in calendar months of March, April, and May?
  • Also, a simple table like following would be very helpful:

Month	Backlog on day 1	Articles created	Articles reviewed	Backlog on last day	Articles deleted	Speedy	ProD	AfD
March	Number
April	Number
May	Number

—usernamekiran(talk) 21:30, 16 June 2017 (UTC)

• @MusikAnimal: could you please tell me where can I get this data? I can put it in the table on every 2nd day of a month manually. —usernamekiran(talk) 22:13, 19 June 2017 (UTC)

• @Usernamekiran: "Articles reviewed" can be found at Wikipedia:New pages patrol/Analysis and proposal#Reviewer participation. For the others, it depends on exactly what you're looking for. So for "articles deleted", do you mean the number of articles that were deleted in that month, or the number of articles that were created in that month that ended up being deleted? The latter will eventually be answered by User:MusikAnimal (WMF)/NPP analysis, along with a Speedy/PROD/AfD breakdown. However I want to give 90 days before including deletion data. This is because from my analysis, most articles that end up being deleted are done so in the first 90 days after they were created. So you'll have to wait for data on April and May. Next, "Articles created" – do you mean articles that went into the backlog, or all articles total? For the latter, try checking Wikistats. For the former, again User:MusikAnimal (WMF)/NPP analysis will give you the answers. "Backlog on the last day" is something I'm not sure how to provide, unfortunately, but I'd like to look more into it and will try to do so soon. MusikAnimal (WMF) (talk) 02:54, 20 June 2017 (UTC)

MrX and Usernamekiran: For both of your stats requests -- we'll see what we can do. I think MusikAnimal has some more stats plans, but we haven't talked about them in detail yet. We're heading into a weekend, so we (probably) won't have any more to say about it until next week. :)

I share the frustration that some folks have been expressing on this page about why it takes so long to get numbers. The big-picture answer is that the WMF doesn't currently keep a dashboard of contribution data, and most of the time each question has to be done as a special request, from people who are supposed to be working on other things. Also, sometimes a phrase that sounds simple, like "Articles created", needs to be defined in terms of all the edge cases -- does it count if a redirect is turned into a page, or it comes from draftspace, or goes to draftspace, or is deleted and then undeleted, etc. The folks on our team really care about this problem, which is why we're spending time thinking and talking with everybody here, and we want to get & provide as much data as we can. Sorry it's taking so long. I'm glad to be part of a community (as staff and as a volunteer) where people care so much about facts. DannyH (WMF) (talk) 01:04, 17 June 2017 (UTC)

Thanks Danny! And no, I am not frustrated at all regarding numbers :-)

Yes, I am aware about the "defining" issue. It gets quite problematic. Thats why i intentionally requested for only "articles created". I mean, i didnt include "articles created by auto-c users", "by non-auto-con, excluding autopatrolled" lol. Tyring to keep it simple, yet useful ;-)

I also think that we can find our solution through this simple method. Lets hope for the best. —usernamekiran(talk) 02:39, 17 June 2017 (UTC)

@RileyBugz: There is already a way to grant the confirmed, and even extended confirmed flags at WP:RFR. But then again, it will be too much workload. Redirecting new/unexperienced users to request page, too many requests to be handled for admins. And also, how to identify good faith users? By the time there is enough evidence, they are already auto-confirmed.

For your other question: there are still a lot of experienced editors who arent aware about NPR flag, and there are a lot of editors who monitor new pages proficiently, but dont have the flag. I (always) say, give these good guys the NPR flag. —usernamekiran(talk) 20:04, 19 June 2017 (UTC)

Well, autoconfirmed isn't even a deal, much less a big one. I think that admins who are patrolling and notice a new user who is creating a ton of good articles should just give out confirmed status to them. If the editors turn out to be sleepers or whatever, so what? Blocking them won't take long. I do agree with you that people with enough experience should also be give the reviewer flag. RileyBugz^会話_投稿記録 20:10, 19 June 2017 (UTC)

@RileyBugz: erm... In that case the user in question would be most probably an autoconfirmed user. 10 edits/5 days. I think all the policies are good currently. All we need to do is to get some active editors in the NPR user group. —usernamekiran(talk) 20:47, 19 June 2017 (UTC)

A tale of three women

Myself, I have auto-patrolled status and so am able to create a steady stream of new articles without much interference but new editors have much more difficulty. Here are three cases which I have noticed recently:

1. Ashley Hannah was one of a large group of students who attended an outreach event at Imperial College recently. She was bright and enthusiastic and wanted advice about how to add an image to the subject that she was working upon – Sue Gibson (chemist). I showed her a way of doing this but it wasn't working. The feedback from the interface wasn't clear but my impression is that this was because she wasn't autoconfirmed and so file uploading was blocked. I could have asked her to make 10 edits but, to be autoconfirmed, four days have to elapse and she had just created her account at this one-day event. So, I wasn't able to resolve her difficulty and this was quite frustrating. She got some acclaim for her first efforts but will have been left with the impression that adding an image to Wikipedia is significantly more difficult than in Instagram, Snapchat or Twitter (the most popular apps for teens). Why do we have a four-day cooling-off period when this is so clearly an obstacle for outreach events?

2. Henrietta999 is an experienced author who spoke at another recent editathon. Her first article on Wikipedia was Margaret, Lady Moir and that went reasonably smoothly but that was perhaps too smooth as I have the impression that she'd have liked more feedback, especially some appreciation for the effort. Her second article, Margaret Dorothea Rowbotham, has not been going so well because it was found by a new page patroller who has been nit-picking and tag-bombing. This has not been well-received and the editor has withdrawn from the topic. Rather than collaboration, we have conflict – a familiar difficulty throughout Wikipedia.

3. Carolineneil has been creating a stream of impressively erudite chemistry articles from Asymmetric addition of dialkylzinc compounds to aldehydes to Use of pi,pi, CH-pi and pi-cation interactions in supramolecular assembly. These have been going through AfC and the results seem to have been reasonably productive. But the editor is quite introverted here and is getting some stick at ANI for not communicating and there's currently a proposal to block them to attract their attention. Are they unfamiliar with our ways of communicating, indifferent to them or repelled by them? Should we be looking a gift horse in the mouth?

These examples seem to confirm what was said in Encyclopedia Frown

“The encyclopedia that anyone can edit” is at risk of becoming, in computer scientist Aaron Halfaker’s words, “the encyclopedia that anyone who understands the norms, socializes him or herself, dodges the impersonal wall of semiautomated rejection and still wants to voluntarily contribute his or her time and energy can edit.”

I support the WMF's recommendation that we should try to keep the project open and accessible. Andrew D. (talk) 10:19, 18 June 2017 (UTC)

OK, Andrew, but what you are talking abut is a special environment, an editathon or outreach event that are attended by genuinely motivated people. What you are avoiding accepting is that 76.1% of the 800 articles a day by new users are correctly deleted. You are talking about exceptions, we're talking about our experience of hundreds of hours of patrolling this rubbish. Do some yourself and you'll understand - Experienced patrollers don't generally flag the wrong stuff for deletion, and if they do, an admin will decline the CSD. Part of the problem is that the community still insists that every newbie and his dog is allowed to tag new pages, and even if an admin puts it right, the damage is already done. Don't get me wrong either, I've facilitated on editathons too. The other part of the problem is that Halfaker’s words ring true but the Foundation refuses to give us the tools that would prevent it from happening. Kudpung กุดผึ้ง (talk) 12:44, 18 June 2017 (UTC)

In the first case, Wikipedia:Requests for permissions/Confirmed seems to be the appropriate path for supervised users at events like this. Organizers of events should keep this in mind and alert administrators to be on hand to promptly approve these requests.

The other two are Wikipedia culture issues that don't seem to have a user permissions component and so don't directly relate to this discussion. It is possible, however, that with less noise from unconfirmed users, confirmed users will be treated (slightly) better. ~Kvng (talk) 13:08, 18 June 2017 (UTC)

@Andrew Davidson: With regards to the last case, by 'reasonably introverted', you mean that the editor in question has not edited her talk page in the two years that she has been on Wikipedia. Because of this, she submits sandbox drafts to AFC even after the same draft has already been accepted into article space and never returns to improve declined drafts, which means much time is wasted by AfC reviewers who take time to leave individualised reviews, often seeking out help from the Chemistry Wikiproject on her behalf. jcc (tea and biscuits) 13:30, 18 June 2017 (UTC)

Andrew Davidson We seem to have similar experiences. I volunteer at edit-a-thons and am often horrified by how we treat new editors. I think your examples demonstrate three distinct issues:

• • The user experience of Wikipedia is terrible. I've heard arguments that the Visual Editor encourages incompetent editors to contribute, and we that we should only teach people how to write wikitext (which we need to teach them anyway or they won't be able to use a talk page). We might as well require that Wikipedia edits must be made in Malbolge, just to keep the riff·raff out. I hate Wikipedia's User Experience, and so does every single new user. It is absolute shit. I'd rather write my edits in Perl with Vim on a command-line than this. Wikipedia's hostility to new editors is instrumentalized in its user interface.
  • Collaboration is hard, and often not desirable in the initial phases of article development. Wikipedia's concept of continuous development of articles only works if contributors are ready to accept that their articles will be ruthlessly edited. They need a button that they understand to mean "I'm ready to let the community rewrite everything I just wrote", not one that says "Save page" or "Submit your draft for review". If an editor writes "I need the freedom to write without interference", then that article should have been sent to draft space. We've been asking for such a feature. I have page mover rights, and committed to not moving pages to draft without a new editor's consent and find that asking for permission does not work at all. The users who need it most don't even have to skills to respond to a question on their talk page. Move to draft would prevent a lot of grief. New users must be told to start in Draft, for their own benefit and ours.
  • Communication is essential to collaboration. It's sad to see a desperate attempt to communicate with a new user end in a (request for) a block, just to get their attention, but really; is it too much to ask to talk to the rest of us? All new editors must learn how to communicate before they do anything else. I think of myself as one of those editors who understands the norms, socializes himself, dodges the impersonal wall of semiautomated rejection and still wants to voluntarily contribute his or her time and energy. A big part of outreach for me is convincing people that doing these things is worthwhile and rewarding. Dissuading new editors from tackling the most difficult thing and instead steering them towards attainable short-term goals benefits everyone, reviewers and new editors alike. Mduvekot (talk) 13:44, 18 June 2017 (UTC)

Concerning images, one can upload images on Commons right after the registration. That would not work if the image is not acceptable for Commons, for instance, for fair-use images, but these are indeed difficult for new users, and, in addition, can not be added to articles on living people.--Ymblanter (talk) 15:55, 18 June 2017 (UTC)

I'm not gonna lie and say that I haven't considered what combination of threats, bribery, and groveling would convince someone like the Green brothers to put together something like this video, but instead something that would give a crash course (pun intended) on things like OR, V, NPOV, etc. TimothyJosephWood 16:32, 18 June 2017 (UTC)

Think of the children

Here's some more recent examples. They start with a case that I just came across while patrolling. The article was children in emergencies and conflicts and was at risk of imminent deletion because it had been proposed for deletion more than seven days before. I did not hesitate to remove the prod because this is quite a notable topic – it's easy to find entire books about it – and its treatment in this case was quite scholarly and well-supported by citations. After removing the prod, I looked to see who had placed it. This was not a vandal or inexperienced editor, but DGG – a veteran patroller and illustrious member of Arbcom. Now, my view is that DGG's action was quite contrary to the WP:PROD process which states clearly that it is for "for uncontroversial deletion. ... PROD must only be used if no opposition to the deletion is expected". It seemed quite inappropriate in this case because DGG said himself, "I don't know if it is fixable." It seems remarkable that such an experienced editor could act like this and so we seem to have the problem of quis custodiet ipsos custodes?. I caught this one but how many other respectable, substantial articles are being casually deleted in this way?

The creator of this article, A.mart82, is working with a Wikimedian in Residence on such topics in partnership with UNESCO. They are a new editor and notice that they created other substantial articles such as child development in Africa just one day after starting editing and so before they were autoconfirmed. Notice also that their talk page just contains templated warnings, including other deletion proposals. Isn't this a remarkably bitey way to treat someone who is working with one of the foremost cultural organisations on the planet? My view is that the WMF should encourage and support such work. If the NPP is unable to treat such work with consideration and respect, then the WMF should perhaps limit their powers rather than augmenting them?

Of course, I might be missing some aspect of this and so I encourage the parties involved to explain themselves and their actions. This will then fully inform our discussion. Andrew D. (talk) 22:55, 24 June 2017 (UTC)

@Andrew Davidson: DGG explained himself and you deliberately took his comments out of context. The article in question reads like an essay and an OR-fest. I don't know that I would have PROD'd it but your comments here are inappropriate and since you seem to need demonstration for BITEy behavior, these are my contributions. Also, don't make an appeal to authority (claiming UNESCO) because that's a cognitive bias. You ought to get that looked at, too. Chris Troutman (talk) 23:18, 24 June 2017 (UTC)

(edit conflict)This seems for the most part to be a reprinting of the two "sources", which is fine by copyright standards, since they're appropriately licensed, and is certainly within the realm of the things that the orgs themselves are want to publish, but it's not clear that there is any real purpose to the article, since it would be difficult to exclude very much of anything with anything other than original research. The topic is just entirely overly broad. It's a bit like Cars in deserts or Dogs in urban environments. There's sources for sure, but it's not entirely clear that the topic isn't mostly arbitrary, and thus WP:OR.

My instinct is that regardless of what the WMF says, we shouldn't have volunteers from UNESCO using UNESCO publications nearly verbatim, to write about things UNESCO is centrally involved in, and then publishing in in WP's voice. There's pretty serious WP:COI written all over that. TimothyJosephWood 23:20, 24 June 2017 (UTC)

I've raised this issue at WP:COIN... mostly because the whole thing just feels kindof dirty. But besides that, I don't think this page is really the right venue. TimothyJosephWood 02:08, 25 June 2017 (UTC)

This is a pervasive problem with non-profits, whose articles tend to be just as promotional as other organizations, but we have more difficult seeing this because of our usual sympathies with their goals. DGG ( talk ) 05:42, 25 June 2017 (UTC)

The answer to quis custodiet ipsos custodes? in this case is WP:PRODPATROL. Thankless work and therefore understaffed. Also, administrators who close PRODs are supposed to assess them before deleting but there is no way a non-administrator can check their work here. ~Kvng (talk) 13:42, 26 June 2017 (UTC)

I thought we were past the point of arguing whether we have a culture problem for new editors on Wikipedia. I haven't seen anyone try to argue otherwise recently. What I've seen is claims that it has to be this way to defend the quality of the encyclopedia or that there's a problem but we don't know how to fix it. ~Kvng (talk) 13:46, 26 June 2017 (UTC)

This study is poorly done and misleading

There are many controversial claims in this study that are not backed up by any evidence:

1. Adding more reviewers to this system will not make it work better. This is a ludicrous statement that is not backed up by evidence. Here's a thought experiment: keep the rate of article creation the same, but add 1 million reviewers to the system. Do you think the backlog would become more manageable? Yes, it would. Therefore, this claim is false.
2. Reducing the number of pages entering this system will not make it work better. Like the first claim, this one is also false. Same thought experiment: if we reduced the number of pages entering the system to one per day, then the system would be far more sustainable than it is today. Therefore, this claim is obviously false.
3. To properly triage the Time-Consuming Judgment Calls, you need a subject matter expert. This claim is also unsupported by any evidence, and is patently false. Reviewing articles does not require subject matter expert because no one is reviewing the accuracy of content within the articles (which is when a subject matter expert would be needed). Reviewers don't even need to understand what the article is about. Reviewers just make sure that the article isn't eligible for deletion, and then perhaps put some tags on the article if there are glaring problems like no references. There has never been a need for subject matter experts in the reviewing process, and trying to insert that requirement into the process would make it far less sustainable than it is now.
4. The backlog is not made up of bad pages; it's made up of marginal pages that are time-consuming to review. Firstly, there is no evidence to support this claim (except for a few cherry-picked articles that are discussed in this study). Secondly, even if it were true, if we were able to get rid of the hundreds of terrible articles that are created by non-autoconfirmed users every day, then the pool of reviewers would have more time to spend on the marginal pages. The assumption made in this study is that reviewers are too lazy or uninterested to work on the marginal articles, when the reality is likely to be that they are overworked by the torrent of shit articles and don't feel they have the time to spend on the marginal articles.
5. The only sustainable way to manage the backlog is to reinstitute the expiration date. This is the most bizarre claim of all. Somehow, the author makes the assumption that any articles that are still in the backlog after 30 days must be marginal articles that have the potential to be great. He believes there is no way that terrible, speedy-deletable articles could ever make it through undetected for 30 days. How can this possibly be considered a solution to the problem? Reinstating the expiration date is only a means of ignoring the problem. (Hint: the problem isn't the backlog, it's the deluge of awful articles being input into the system by clueless or malicious users.)

Secondly, no serious attempt is made to judge the quality of articles created by non-autoconfirmed users. Per MusikAnimal's subsequent analysis, we see that a full 3/4 of articles created by non-autoconfirmed users are eventually deleted. Why are we fighting so hard to keep allowing new users to create this crap and publish it in mainspace? Restricting non-autoconfirmed users from creating articles would not only significantly reduce the number of articles that need to be reviewed, but it would also vastly increase the average quality of articles that reviewers are reviewing. This would change the whole process of reviewing. Instead of it being the current war against the relentless diarrhea emanating from the depths of humanity, it could become a more constructive process with an emphasis on improving articles and educating serious editors.

Thirdly, remember that ACTRIAL was designed to be a temporary trial. There is only one way to definitely determine the effect of ACTRIAL on the backlog, article quality, and editor retention: implement the trial for a temporary period, and see what happens. We can do more studies and argue about it until the cows come home, but we won't come any closer to understanding the true effect unless we actually do the trial. Even if ACTRIAL turns out to be a miserable failure, it would only be a failure for a short time, and the damage that it could do to the project would be minimal. ‑Scottywong| babble _ 06:56, 6 July 2017 (UTC)

I'm in agreement. What's the harm of trying? A 6 month trial isn't going to hurt the pool of new articles any more than unchecked vote canvassing is today. TimTempleton ^{(talk) (cont)} 01:02, 7 July 2017 (UTC)

Timtempleton, the WMF has agreed to implement ACTRIAL in principle and the current conversation around the specifics is going on at WT:NPPAFC. Scottywong's critique here is spot on, and he has also commented at the discussion on the other page. TonyBallioni (talk) 01:42, 7 July 2017 (UTC)

Yes, the discussion that we've had here and on other pages, and MusikAnimal's round of stats, convinced us that we should take ACTRIAL seriously and help to make it successful as an experiment. I still think that the core problem with the backlog is the change in scope, but I'm interested in testing all of these ideas and assumptions. -- DannyH (WMF) (talk) 23:34, 7 July 2017 (UTC)

DannyH (WMF), I still think you are wrong about the scope having changed, and that yours is a subjective interpretation of the chart I published. The only thing that has changed is the still failed attempt to prevent totally inexperienced users from letting spam and junk through while flagging the articles of good faith users for deletion. The entire objective of my high number of New Page Reviews - that has contributed to the recent slight drop in the backlog - is in smoking these people out and telling them to stay out of serious maintenance areas, while encouraging them to do some vandalism patrolling instead. As you prefer stats, if I do 50 patrols in a session, I've usually chased at least one patroller away. This problem goes back years, long before I started all these discussions in October last year. Kudpung กุดผึ้ง (talk) 03:24, 8 July 2017 (UTC)