Finish Edit Check measurement plan proposal
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ppelberg
	Dec 22 2022, 10:01 PM

Description

This task involves the work with finalizing a proposal for the measurement plan that will guide the experiments and analyses the Editing Team will run and complete to evaluate the impact of the Edit Check project.

NOTE: we will consult with volunteers about this measurement plan as part of T333709.

Draft

Edit Check/Measurement Plan

Process

Step	Description	People Responsible	Status
1.	Draft measurement plan (e.g. project goals, hypotheses, research questions, experiment plan, etc.)	@ppelberg	✅ done
2.	Review measurement plan draft + propose methods/metrics for evaluating hypotheses and answering research questions	@MNeisler + @nayoub	✅ done
3.	Review measurement plan as a team	Editing-team	✅ done
4.	Finalize measurement plan	@MNeisler + @ppelberg	✅ done
5.	~~Publish measurement plan on mediawiki.org~~	@ppelberg	Publication and consultation will happen in T333709

Done

Each step in the ===Process outlined above is completed

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T265163 Create a system to encode best practices into editing experiences
Open		Trizek-WMF	T331946 [RELEASE TICKET] Make Edit Check (references) available to all newcomers at all Wikipedias
Resolved		None	T331947 [RELEASE TICKET] Make Edit Check MVP available to all newcomers at Partner Wikipedias
Resolved	Release	ppelberg	T333709 [CONSULT] Collaborate with volunteers to define Edit Check key results
Resolved		ppelberg	T338907 [RELEASE TICKET] Make Edit Check MVP available as Beta Feature (all wikis)
Resolved		None	T332646 [Deployment] Make Edit Check MVP (mobile + desktop) available for testing at test.wikipedia.org
Resolved		ppelberg	T328944 Build Edit Check MVP (references)
Resolved		Ryasmeen	T324735 Define instrumentation needed to evaluate how people engage with Edit Check
Resolved		ppelberg	T325838 Finish Edit Check measurement plan proposal
Open		None	T338962 Invite experienced volunteers to experiment with Edit Check MVP
Open		None	T339165 Create config that enables you to set default editing mode when creating a patch demo
Resolved		Esanders	T340086 Refine Edit Check reference heuristic
Resolved		dchan	T340088 Trigger reference check only on pure insertions (remove.length === 0)
Duplicate		None	T340704 Define the account attributes for which Edit Check should be made available
Resolved		Esanders	T341307 Ready Edit Check for production
Open		None	T327959 Create ways for volunteers to audit, configure, and evalute Edit Check
Resolved		ppelberg	T330112 Implement reference check for on-wiki configuration
Resolved		ppelberg	T330807 [SPIKE] Investigate technical approaches for making Edit Check configurable on a per project basis
Resolved		Esanders	T344962 Make placement of reference within sentence configurable based on local wiki style guide
Resolved		DLynch	T346949 Implement edit check configuration setting to restrict sections it'll apply within
Open		None	T347775 Implement edit check configuration setting to specify categories it'll apply within
Open		None	T342189 Create a way for volunteers to see the edits an Edit Check was activated within
Open		None	T342460 [SPIKE] Investigate edit tag additional data storage approach
Open		None	T373949 Clarify the meaning of the editcheck-references-activated tag
Resolved		Esanders	T342462 Introduce a new tag to identify edits the reference check is shown within
Open		None	T342478 [SPIKE] Investigate performance of creating custom views of the revision log
Open		None	T373690 Diffs are tagged as "Edit Check (references) activated" while users haven't added any footnote
Open		None	T343166 Enable volunteers to evaluate reference check impact
Resolved		Esanders	T343168 Define where/how to report Edit Check false positives
Open		None	T349119 Turn Edit Check tags into links
Resolved		Esanders	T349264 Link editcheck-reference-activated tag to Edit Check false positive reporting page
Open		None	T372927 Enable volunteers to configure Edit Check using Community Configuration
Resolved		ppelberg	T342157 Conduct usability testing of initial edit check user experience (desktop)
Invalid		None	T342471 Re-introduce an edit tag to identify edits that meet the Edit Check heuristic
Resolved		ppelberg	T329985 [SPIKE] Determine Edit Check deployment phases
Open		None	T342311 [Edit Check] Edit check should not be triggered for the same edit that has been responded to with reason to add a reference

Event Timeline

ppelberg created this task.Dec 22 2022, 10:01 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 22 2022, 10:01 PM

ppelberg assigned this task to MNeisler.Dec 22 2022, 10:01 PM

ppelberg moved this task from Incoming to Doing on the Editing-team (Kanban Board) board.

ppelberg moved this task from To Triage to Triaged on the VisualEditor board.

nshahquinn-wmf added a project: Product-Analytics.Dec 23 2022, 12:13 AM

nshahquinn-wmf removed a subscriber: Product-Analytics.

MNeisler triaged this task as Medium priority.Jan 17 2023, 5:08 PM

MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.

VPuffetMichel added a project: EditCheck.Jan 18 2023, 3:59 PM

MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Feb 2 2023, 6:05 PM

@ppelberg
I've completed my review of the Edit Check Measurement Plan and proposed some quantitive metrics and evaluation methods.

Currently reassigning this task to you for review.

ppelberg edited parent tasks, added: T328944: Build Edit Check MVP (references); removed: T265163: Create a system to encode best practices into editing experiences.Feb 20 2023, 10:10 PM

ppelberg added a parent task: T324735: Define instrumentation needed to evaluate how people engage with Edit Check.Feb 20 2023, 10:20 PM

ppelberg removed a parent task: T328944: Build Edit Check MVP (references).

ppelberg mentioned this in T324735: Define instrumentation needed to evaluate how people engage with Edit Check.

In T325838#8622340, @MNeisler wrote:

@ppelberg
I've completed my review of the Edit Check Measurement Plan and proposed some quantitive metrics and evaluation methods.

Currently reassigning this task to you for review.

Alright! @MNeisler: I've gone in and responded to the open comments within the measurement plan draft.

By the time you and I finish meeting on Wednesday (22 Feb), I think we'll be in a good position to share this with the team for review.

ppelberg updated the task description. (Show Details)Feb 20 2023, 10:42 PM

ppelberg renamed this task from Publish Edit Check measurement plan to Finish Edit Check measurement plan proposal.Mar 31 2023, 5:15 PM

ppelberg added a parent task: T333709: [CONSULT] Collaborate with volunteers to define Edit Check key results.

ppelberg updated the task description. (Show Details)

The measurement plan has been reviewed and updated to ensure it reflects the leading indicators from the pre-mortem and was used as the basis for the proposed instrumentation documented in the instrumentation spec and mobile annotated UX mockup.

This and the instrumentation requirements have been shared with the Editing team via Slack.

Next step: Complete final review with @ppelberg

MNeisler updated the task description. (Show Details)May 12 2023, 2:48 PM

ppelberg moved this task from Doing to Ready for Sign Off on the Editing-team (Kanban Board) board.Jul 26 2023, 12:25 AM

In T325838#8847588, @MNeisler wrote:

The measurement plan has been reviewed and updated to ensure it reflects the leading indicators from the pre-mortem and was used as the basis for the proposed instrumentation documented in the instrumentation spec and mobile annotated UX mockup.

This and the instrumentation requirements have been shared with the Editing team via Slack.

Next step: Complete final review with @ppelberg