I took a look at the codebase and regarding prometheus:

• We are using latest service runner which is compatible with prometheus
• All the custom metrics are made with makeMetric are prometheus compatible

Looks ok in nodejs level too:

Ok, i think thats good enough. I will double check in the image locally that nodejs actually trusts the exact cert just to be on the safe side and close the ticket.

Update from debugging:

I am reverting production envs only and leave staging in case SREs need to debug the actual running pod.
FYI we tried to reproduce it locally but we didn't get any failures.

I run a rolling restart in k8s. Regarding the chromium parameters we already pass the --single-process. I am taking a look at what the other parameter does in detail.

The patch above creates an image using:

• Latest mapnik built as a step in blubber config
• Node bindings build as part of blubber
• Kartotherian uses latest mapnik

Looks OK to me.

We currently have hewiki with caching enabled. It looks like our mechanism (which is the same logic we had on restbase) works as expected. Closing this for now.

Thanks @Aklapper leaving it open until gitlab 2fa is also reset.

Hey @Aklapper unfortunately i couldn't reset it with only my old 2fa (it needs both OTPs). Can we arrange a video call to reset my 2fa? (cc'ing my manager @SLopes-WMF )

Done, I will keep an eye on the logs.

I suspect that the issue is that we don't close or somehow we end up in a sitation with stale browser instances. Given the level of traffic/support of the pdf service would it be enough to just restart the service ?

@Aklapper I found my old 2fa that is already registered in my phabricator (i have 2 registered in my profile, i assume i never deleted the old one). Would that be of any use to prove my identity ?

gitlab

You may have recovery codes? If not, https://docs.gitlab.com/ee/user/profile/account/two_factor_authentication_troubleshooting.html#generate-new-recovery-codes-using-ssh could be an option?

You are right, that could be an option I still have them

• I managed to authenticate to gerrit/horizon so that works fine.
• I still need 2fa reset for phabricator/gitlab/npm

Hi @elukey thanks for the update. Regarding building the image, I can give it a try to use blubber but i suspect that its not gonna work because the build instructions are more complicated. Do we another way to define images and upload them to docker-registry?

This metric has been superseeded by ParserOutputAccess.<key>

This stats key has been replaced by htmloutputrendererhelper. Closing this one as invalid since we do have another ticket for the latest stats key.

Codesearch returns no results: https://codesearch.wmcloud.org/search/?q=parsoidhelper&files=&excludeFiles=&repos=
Closing this one as invalid.

After a discussion with @Seddon it looks like hewiki is a good alternative.

PR here: https://github.com/wikimedia/restbase/pull/1351

@Seddon informed me that ptwiki is used for some experiments so it not might be the best idea to use this for testing the new stack without restbase.

This is fixed here:
https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/1062375