Background
We would like to check that the events are from the A/B test schema are logged as expected
Acceptance criteria
- QA A/B test schema post-deployment
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | ovasileva | T333180 [EPIC] A/B test Zebra #9 prototype | |||
| Resolved | jwang | T335972 Launch content separation (Zebra #9) A/B test | |||
| Resolved | jwang | T336299 QA A/B test schema for Zebra #9 A/B test |
Event Timeline
Comment Actions
QA Summary
Have QAed the events in event.mediawiki_web_ab_test_enrollment schema between 2023-06-02 and 2023-06-19, the instrumentation captured the events as expected.
- Bucketing is balanced across wikis
- 0.12% of sessions are assigned to both control group and treatment group. They should be excluded in analysis.
| wiki | sessions in both test groups |
|---|---|
| enwiki | 1142 |
| fawiki | 81 |
| frwiki | 130 |
| hewiki | 40 |
| kowiki | 25 |
| ptwiki | 80 |
| srwiki | 10 |
| thwiki | 15 |
| trwiki | 38 |
| viwiki | 27 |
- 466 sessions are from bot users, should be exclude in analysis.
More details are in analysis report.