HomePhabricator
Phame Blogs Wow. So wikimedia. Such quality. Many testing. Very team. J317
Refactoring WebdriverIO Tests From Sync to Async Mode

Prehistory

It's hard to believe it, but there were Selenium tests in Wikimedia repositories even before I joined the Wikimedia Foundation. (12 years ago, in 2012.) In MediaWiki Core, Selenium tests (in PHP) existed from 2010 until 2013.

When I started working for the Foundation, Selenium tests (in Ruby) existed in the qa/browsertests repository. The repository was active from 2012 until 2014. Tests from the qa/browsertests repository moved to the MediaWiki Core repository in 2014. They lived in tests/browser folder until 2017. (T139740)

History

In 2017 we decided to rewrite Selenium tests in Javascript.

Since then we had a few major framework updates:

  • Port Selenium tests from Ruby to Node.js (2016 - 2023) (T139740)
  • Evaluate WebdriverIO replacements for our browser automation framework (2020) (T247835)
  • Upgrade WebdriverIO to v6 in all repositories (2020-2021) (T247844)
  • Evaluate Microsoft Playwright as replacement for our browser automation (2020-2021) (T262682)
  • Upgrade WebdriverIO to v7 in all repositories (2021-2023) (T274579)
  • Switch to spec reporter (2021-2024) (T280331)
  • Refactor WebdriverIO tests from sync to async mode (2020-2024) (T256626)

Async

The sync to async migration is just one of the major updates to the framework. But, it's an important one.

In 2021 WebdriverIO announced they are deprecating the sync framework. In 2020 I created an epic task (T256626) to track the migration in more than thirty repositories. (I'm not sure why I created the task before the announcement. Maybe I knew the change was coming in advance.) I had resolved the task in 2024. It took us a while.

The migration project started with an internship (2021-2022). During the internship we did the majority of the migration. The internship lasted only a few months. We tried to migrate as many repositories as possible. We managed to migrate repositories that didn't give us a lot of trouble. The few remaining repositories were hard to migrate.

This monster Gerrit search finds most commits related to this project. There are 92 commits. 58 merged, 34 abandoned.

We started strong in 2020, but we were not able to finish the migration for several repositories. The main problem was technical. We could not set up a working development environment for some repositories (T344682).

At the Wikimedia Hackathon 2024 I talked with several people from Wikimedia Deutschland about the async project. (At least with @Lucas_Werkmeister_WMDE and @WMDE-Fisch.) After looking at the tracking Phabricator task (T256626), they said their repositories are already migrated to async, but nobody closed the subtasks.

Documentation

After closing relevant subtasks, the only thing left to do was to update the documentation (T364268). All of our documentation was using sync syntax. After the hackathon I started updating the documentation.

After looking at the documentation I realized that there are several pages with a lot of code on them. All of the code was now broken. If we update the code now, it will become broken the next time we make a change to our framework. Nobody would know that the sample code on a wiki page was broken. That is, until somebody tried to use it.

To solve the problem of broken code on wiki pages, we have decided to move all sample code to MediaWiki Core (T368535). That way, we would be reminded (actually, forced) to update the sample code at the same time we update the framework. If we update the framework and not the sample code, our continuous integration would break.

I had a lot of help from people on my team. We have an office hours meeting almost every day. I was working on the documentation during the day and asking for feedback during the meeting. The process worked great.

When all the code was moved from the wiki pages to MediaWiki Core and updated to use async syntax, we copy/pasted the code back to the wiki pages. That's the last remaining step that is still manual. Nobody on my team knew how to embed a file in a wiki page. If you know how to do it, please let me know.

Updating the documentation finished the async migration project.

Languages and Tools

While doing some online archaeology for this blog post, I stumbled upon Selenium tests in Wikimedia projects written in PHP (search), Python (search) and Ruby (search).

Also, Selenium is not the only tool we use for end-to-end testing. There's Cypress (search), Puppeteer (search), Playwright (search) and Nightwatch (search).

Future

We're currently working on a few projects that will further improve our Selenium testing framework. There are plenty of future improvements we plan to do. See Selenium Framework Updates Timeline section at the end of the article for more information.

A notable project is T344682: Run Selenium tests _targeting available local and remote testing environments. We need testing environments for more than forty repositories. Unfortunately, that's not a solved problem. There are many options, but I'm not sure that any of them would work with all repositories. A good solution would be to find at least one development environment where a particular repository would work. If that means we have to use multiple environments to get all repositories working locally, I'm fine with that. I'm not happy with that, but even that would be an improvement over the current situation. For some repositories we don't have a working local development environment at all.

Async Migration Timeline

Selenium Framework Updates Timeline

Past

  • Port Selenium tests from Ruby to Node.js (2016 - 2023) (T139740)
  • Evaluate WebdriverIO replacements for our browser automation framework (2020) (T247835)
  • Upgrade WebdriverIO to v6 in all repositories (2020-2021) (T247844)
  • Evaluate Microsoft Playwright as replacement for our browser automation (2020-2021) (T262682)
  • Upgrade WebdriverIO to v7 in all repositories (2021-2023) (T274579)
  • Refactor WebdriverIO tests from sync to async mode (2020-2024) (T256626)
  • Switch to spec reporter (2021-2024) (T280331)

Present

  • Run Selenium tests _targeting available local and remote testing environments (2023-) (T344682)
  • Replace Node.js Assert library with WebdriverIO Expect library (2022-) (T325740)

Future

  • Upgrade WebdriverIO to v8 in all repositories (2022-) (T324685)
    • Use import instead of require (2024-) (T373125)
  • Upgrade WebdriverIO to v9 in all repositories (2024-) (T363704)
  • Implement mediawiki core selenium test suite in popular alternatives (2022-) (T325552)
  • Find flaky Selenium tests (2021-) (T291254)
Previous Post
Moving performance testing tools out of AWS
Next Post
Google Web Vitals, 75th percentile and why every milliseconds count on Wikipedia
Written by zeljkofilipin on Oct 15 2024, 9:47 AM.
Senior Software Engineer in Test (Contractor)
Projects
None
Subscribers
WMDE-Fisch, Lucas_Werkmeister_WMDE
Tokens
"Party Time" token, awarded by abi_.

Event Timeline

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits
  NODES
INTERN 3
Note 1
Project 8