Page MenuHomePhabricator

Wikisource Export: Migrate existing DB system to use Doctrine ORM
Closed, ResolvedPublic5 Estimated Story Points

Description

As a Wikisource Export developer, it would be preferred if our database used Doctrine ORM to be consistent with Symfony standards and allow for easier management of the schema.

Background: Currently WS Export is using it's own system to create and update the database (which only logs exports at the moment). While not strictly necessary, it is better to move to Doctrine ORM, complete with migrations, to stay more consistent with Symfony standards and more easily allow us to make changes to the schema. This would be especially beneficial if/when we tackle the job queue (T253283). Finally, having our querying go through Doctrine will allow for easier caching if/when we migrate the phetools queries to WS Export (T257886).

Acceptance criteria:

  • Convert the existing CreationLog class to use Doctrine ORM
  • Querying itself does not necessarily need to use the query builder but it could.

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptOct 21 2020, 9:50 PM
ARamirez_WMF set the point value for this task to 5.Dec 10 2020, 6:19 PM
ARamirez_WMF moved this task from Needs Discussion to Up Next (June 3-21) on the Community-Tech board.

Realized I never posted the PR here: https://github.com/wsexport/tool/pull/291

That has been merged. Note to QA: basically nothing user-facing should have changed. We did add an id column to the books_generated table. The README has been updated as the installation steps have changed slightly. So I guess the main thing we're testing is if anything broke as far as storing exports in the the books_generated table.

dom_walden added subscribers: ifried, dom_walden.

...So I guess the main thing we're testing is if anything broke as far as storing exports in the the books_generated table.

As far as I can see from looking at the database on wsexport-test, I could not see any problems with the accuracy or formatting of the data inserted into the database table.

I also tested statistics, which reads from the books_generated table. All the stats were accurate as far as I could tell.

@MusikAnimal Following the Docker steps in the README.md. When I ran docker-compose exec wsexport ./bin/console doctrine:database:create I got:

SQLSTATE[HY000]: General error: 1007 Can't create database 'wsexport'; database exists

I am assuming because the mariadb image in the docker compose already creates the wsexport database.

It works if instead I run docker-compose exec wsexport ./bin/console doctrine:migrations:migrate --no-interaction.

Should we change the docker-compose.yml, the README or the code? (@ifried This only affects developers. This is not going to affect production when this change gets pushed there. So I am moving this on.)

This has been merged & has passed QA. It only impacts developers (not production), as noted by Dom. For this reason, I'm marking this work as Done.

MusikAnimal Following the Docker steps in the README.md. When I ran docker-compose exec wsexport ./bin/console doctrine:database:create I got:

SQLSTATE[HY000]: General error: 1007 Can't create database 'wsexport'; database exists

I am assuming because the mariadb image in the docker compose already creates the wsexport database.

It works if instead I run docker-compose exec wsexport ./bin/console doctrine:migrations:migrate --no-interaction.

Should we change the docker-compose.yml, the README or the code?

Thanks for pointing this out. README updated with https://github.com/wsexport/tool/pull/307

  NODES
COMMUNITY 14
Note 3
Project 2