User Details
User Details
- User Since
- Nov 2 2020, 11:59 AM (217 w, 3 d)
- Availability
- Available
- IRC Nick
- dcaro
- LDAP User
- David Caro
- MediaWiki User
- DCaro (WMF) [ Global Accounts ]
Thu, Dec 12
Thu, Dec 12
dcaro added a comment to T381899: Add support for Python 3.13.
dcaro added a comment to T381923: Toolforge Build Service does not support .python-version.
Yep, the documentation shifted and we did not update (yet) the buildpack underneath, this will be solved with the next buildpack upgrade, related tasks:
dcaro triaged T374056: Upgrade python buildpack to v0.17.0 or newer for Poetry support as Medium priority.
Restricted Application added a project to T363854: Upgrade golang buildpack to 1.22: cloud-services-team.
Restricted Application added a project to T353762: Python buildpack does not detect requirements from pyproject.toml: cloud-services-team.
Wed, Dec 11
Wed, Dec 11
dcaro added a comment to T381911: cloud VPS project codesearch or devtools - quota increase request for storage volumes.
Is this considered a lot or not at all compared to what you have available for all of cloud VPS?
Nov 29 2024
Nov 29 2024
dcaro moved T380832: [jobs-api] crashing from Next Up to Done on the Toolforge (Toolforge iteration 16) board.
Nov 28 2024
Nov 28 2024
dcaro moved T379893: [toolforge-weld] read setting from envvars too from In Review to In Progress on the Toolforge (Toolforge iteration 16) board.
dcaro added a comment to T366579: [infra,k8s,monitoring] Add an alert to warn when the prometheus k8s cert is about to expire.
Up and running!
dcaro added a comment to T380127: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble.
dcaro added a comment to T380127: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble.
I think we can add a flag like '--new-builder' or similar, for people to try (the tekton pipeline already allows customizing it, might have to be added to the api too). Then when we are happy, make it the default, and move the flag to --old-builder for people to migrate if they have not yet, and eventually deprecate, we can probably be a bit more relaxed than with the API changes and give tool maintainers a bit more time to migrate.
dcaro added a comment to T379927: Puppet removed "nameserver" line from /etc/resolv.conf.
I tried with:
resolver = Resolv::DNS.new( :nameserver => '127.0.0.1', :raise_timeout_erros => true, )
dcaro added a comment to T380960: kernel error detector: have a way to ignore certain messages.
That would be nice to have. But I would not know how to implement this off the top of my head. How would you implement it?
dcaro added a comment to T379927: Puppet removed "nameserver" line from /etc/resolv.conf.
Nov 27 2024
Nov 27 2024
dcaro added a comment to T371501: Configure DSCP marking for cloudceph* hosts.
A quick search did not find any reference for the mon option on the upstream ceph, but found a commit on a clone:
dcaro added a comment to T380960: kernel error detector: have a way to ignore certain messages.
Another possibility (maybe on top of) would be to be able to acknowledge the errors, for example read a timestamp from a file before which the errors will be ignored (ex. if an issue might happen again, but the current event is not relevant anymore).
dcaro triaged T381011: [functional-tests,deploy,cookbook] Run only selected tests when deploying a component as Medium priority.
dcaro added a comment to T374830: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts such as gerrit.wikimedia.org.
Noting that this is not specific of this project, on tools we can see failures spread throughout the day from apiserver pods on k8s:
root@tools-k8s-control-9:~# kubectl logs --timestamps -n kube-system pod/kube-apiserver-tools-k8s-control-7 | grep timeout
2024-11-26T19:26:14.837618565Z W1126 19:26:14.837267 1 logging.go:59] [core] [Channel #18526 SubChannel #18527] grpc: addrConn.createTransport failed to connect to {Addr: "tools-k8s-etcd-23.tools.eqiad1.wikimedia.cloud:2379", ServerName: "tools-k8s-etcd-23.tools.eqiad1.wikimedia.cloud", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup tools-k8s-etcd-23.tools.eqiad1.wikimedia.cloud on 172.20.255.1:53: read udp 172.16.0.144:54305->172.20.255.1:53: i/o timeout"
...
2024-11-27T15:19:17.460516016Z W1127 15:19:17.460205 1 logging.go:59] [core] [Channel #59647 SubChannel #59648] grpc: addrConn.createTransport failed to connect to {Addr: "tools-k8s-etcd-24.tools.eqiad1.wikimedia.cloud:2379", ServerName: "tools-k8s-etcd-24.tools.eqiad1.wikimedia.cloud", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup tools-k8s-etcd-24.tools.eqiad1.wikimedia.cloud on 172.20.255.1:53: read udp 172.16.0.144:57729->172.20.255.1:53: i/o timeout"
2024-11-27T15:29:19.548211117Z W1127 15:29:19.547805 1 logging.go:59] [core] [Channel #59987 SubChannel #59988] grpc: addrConn.createTransport failed to connect to {Addr: "tools-k8s-etcd-24.tools.eqiad1.wikimedia.cloud:2379", ServerName: "tools-k8s-etcd-24.tools.eqiad1.wikimedia.cloud", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup tools-k8s-etcd-24.tools.eqiad1.wikimedia.cloud on 172.20.255.1:53: read udp 172.16.0.144:56126->172.20.255.1:53: i/o timeout"
2024-11-27T15:35:20.845977552Z W1127 15:35:20.845689 1 logging.go:59] [core] [Channel #60194 SubChannel #60195] grpc: addrConn.createTransport failed to connect to {Addr: "tools-k8s-etcd-24.tools.eqiad1.wikimedia.cloud:2379", ServerName: "tools-k8s-etcd-24.tools.eqiad1.wikimedia.cloud", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup tools-k8s-etcd-24.tools.eqiad1.wikimedia.cloud on 172.20.255.1:53: read udp 172.16.0.144:39192->172.20.255.1:53: i/o timeout"dcaro added a comment to T380833: [harbor] some artifacts and projects seems to have gone missing.
Full list of tools without artifact or with no project in harbor (total of 8, half of them are ours):
dcaro@urcuchillay$ grep '###' out ################ unable to find project tool-mbh ################ unable to find project tool-milhistbot ################ unable to find project tool-nodejs-flask-buildpack-sample ################ unable to find project tool-teddybot ################ unable to find project tool-containers ################ unable to find project tool-lebot ################ unable to find project tool-sample-complex-app ################ unable to find project tool-containers
dcaro added a comment to T380833: [harbor] some artifacts and projects seems to have gone missing.
a quick scan of the harbor images running on the cluster reveals 3 projects with missing harbor project:
dcaro@urcuchillay$ grep '###' out ################ unable to find project tool-nodejs-flask-buildpack-sample ################ unable to find project tool-teddybot ################ unable to find project tool-lebot
dcaro updated the task description for T380959: [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components.
dcaro renamed T380985: [infra,k8s,o11y] Introduce worker checks from [infra,k8s] Introduce worker checks to [infra,k8s,o11y] Introduce worker checks.
dcaro renamed T380892: [infra,k8s,o11y] introduce additional observability for calico and general networking from toolforge: introduce additional observability for calico and general networking to [infra,k8s,o11y] introduce additional observability for calico and general networking.
dcaro renamed T380959: [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components from toolforge: create docs on how to operate the cluster and core components to [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components.
dcaro edited projects for T380832: [jobs-api] crashing, added: Toolforge (Toolforge iteration 16); removed Toolforge.
dcaro updated the task description for T380959: [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components.
dcaro updated the task description for T380959: [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components.
dcaro added subtasks for T380959: [docs,envvars-api,jobs-api,builds-api] create docs on how to operate the cluster and core components: T325172: [builds-api,harbor,builds-builder] user-story 11: I want to know how to debug the service, T325166: tbs: user-story 10: I want to know how to manage the service.
dcaro closed T325174: [builds-builder,harbor,bulid-service,docs] user-story 11: Add section to admin docs on how to debug the service, how to pin-point the failing component and how to get the logs for each of them., a subtask of T325172: [builds-api,harbor,builds-builder] user-story 11: I want to know how to debug the service, as Resolved.
dcaro added a comment to T380832: [jobs-api] crashing.
why no alerts?
dcaro updated the task description for T320284: [jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components.
Nov 26 2024
Nov 26 2024
dcaro added a comment to T380890: jobs-api: Impersonate user instead of loading certs from NFS.
Could it use a service account instead? (would be simpler)
dcaro added a comment to T380503: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54.
Looks good on my side 👍
dcaro added a comment to T380877: Kernel error Server cloudcephmon1004 may have kernel errors.
The first is expected, the second seems harmless too:
https://hetzbiz.cloud/2024/06/11/those-damn-mpt3sas_cm0-messages/
dcaro added a comment to T380877: Kernel error Server cloudcephmon1004 may have kernel errors.
Current errors:
root@cloudcephmon1004:~# journalctl -k -p err -- Journal begins at Tue 2024-11-26 12:46:45 UTC, ends at Tue 2024-11-26 15:12:54 UTC. -- Nov 26 13:11:41 cloudcephmon1004 kernel: x86/cpu: VMX (outside TXT) disabled by BIOS Nov 26 13:11:41 cloudcephmon1004 kernel: mpt3sas_cm0: Trace buffer memory 2048 KB allocated
dcaro moved T362867: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 from In Review to In Progress on the Toolforge (Toolforge iteration 16) board.
dcaro reopened T364870: Q4:rack/setup/install new cloudcephmon hosts as "Open".
Node up and running
dcaro closed T374005: [ceph] install and put in the cluster the cloudcephmon100[1-3] replacements as Resolved.
Last node added
dcaro updated the task description for T374005: [ceph] install and put in the cluster the cloudcephmon100[1-3] replacements.
dcaro closed T364870: Q4:rack/setup/install new cloudcephmon hosts, a subtask of T374005: [ceph] install and put in the cluster the cloudcephmon100[1-3] replacements, as Resolved.
dcaro added a comment to T380827: tools-nfs outage 2024-11-25.
This one never got resolved https://bugs.launchpad.net/neutron/+bug/1868098 :/
dcaro added a comment to T380827: tools-nfs outage 2024-11-25.
An upstream says its a "harmless log message" https://bugzilla.redhat.com/show_bug.cgi?id=1506035
dcaro added a comment to T380827: tools-nfs outage 2024-11-25.
So my theory is maybe a ceph network hiccup?
Forgot to fill up I guess
dcaro added a comment to T380844: 2024-11-26 Toolforge DNS incident.
The current status is stable again, we are still investigating the root causes, but the cluster is up and running.
Nov 25 2024
Nov 25 2024
dcaro closed T378500: [components-api] Add feature flag to disable user endpoints for deployment in tools as Resolved.
I think it's not needed anymore yep
This one had gotten stuck on cloudcephmon1001, restarted the mon process there and went away.
The cable and adapters have been replaced, and the connection is now up and running \o/
dcaro triaged T380706: [components-api, components-cli] deploy-token: separate create from update as Medium priority.
dcaro added a comment to T380703: versions.toolforge.org is down.
There might be a mixture of issues here, as the original error seemed to happen before the network outage (November 25, 2024 at 11:28:51 AM GMT+1 https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/132#note_8339f66545cb4552cd41ab090f629bde43e09aba).
dcaro added a comment to T374830: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts such as gerrit.wikimedia.org.
There is an ongoing network outage on cloudvps (caused by T380174), working on it.
dcaro closed T380283: [components-api] Limit the amount of deployments to (say) 25, a subtask of T362051: [components-api] First iteration of the component API, as Resolved.
Nov 21 2024
Nov 21 2024
dcaro reassigned T380511: [horizon] failing to show the proxies tab for some projects from dcaro to Andrew.
dcaro added a comment to T380511: [horizon] failing to show the proxies tab for some projects.
dcaro added a comment to T380511: [horizon] failing to show the proxies tab for some projects.
:facepalm: I'm adding docker pull there xd
dcaro added a comment to T380511: [horizon] failing to show the proxies tab for some projects.
Yep, we are not running latest:
dcaro added a comment to T380511: [horizon] failing to show the proxies tab for some projects.
This should have fixed it actually:
https://gerrit.wikimedia.org/r/c/openstack/horizon/wmf-proxy-dashboard/+/1091859
dcaro added a comment to T380511: [horizon] failing to show the proxies tab for some projects.
From our code for the special view:
/opt/lib/python/site-packages/wikimediaproxydashboard/views.py:
dcaro added a comment to T380511: [horizon] failing to show the proxies tab for some projects.
The original exception is:
[Thu Nov 21 17:09:40.191562 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] Error while rendering table rows. [Thu Nov 21 17:09:40.191580 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] Traceback (most recent call last): [Thu Nov 21 17:09:40.191582 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] File "/opt/lib/python/site-packages/horizon/tables/base.py", line 1933, in get_rows [Thu Nov 21 17:09:40.191584 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] row = self._meta.row_class(self, datum) [Thu Nov 21 17:09:40.191586 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] File "/opt/lib/python/site-packages/horizon/tables/base.py", line 590, in __init__ [Thu Nov 21 17:09:40.191588 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] self.load_cells() [Thu Nov 21 17:09:40.191589 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] File "/opt/lib/python/site-packages/horizon/tables/base.py", line 640, in load_cells [Thu Nov 21 17:09:40.191591 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] display_name = table.get_object_display(datum) [Thu Nov 21 17:09:40.191592 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] File "/opt/lib/python/site-packages/horizon/tables/base.py", line 1835, in get_object_display [Thu Nov 21 17:09:40.191594 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] display_key = self.get_object_display_key(datum) [Thu Nov 21 17:09:40.191595 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] TypeError: get_object_display_key() takes 0 positional arguments but 2 were given [Thu Nov 21 17:09:40.192967 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] Internal Server Error: /project/proxy/ [Thu Nov 21 17:09:40.192978 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] Traceback (most recent call last): [Thu Nov 21 17:09:40.192980 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] File "/opt/lib/python/site-packages/horizon/tables/base.py", line 1933, in get_rows [Thu Nov 21 17:09:40.192982 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] row = self._meta.row_class(self, datum) [Thu Nov 21 17:09:40.192984 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] File "/opt/lib/python/site-packages/horizon/tables/base.py", line 590, in __init__ [Thu Nov 21 17:09:40.192985 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] self.load_cells() [Thu Nov 21 17:09:40.192987 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] File "/opt/lib/python/site-packages/horizon/tables/base.py", line 640, in load_cells [Thu Nov 21 17:09:40.192989 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] display_name = table.get_object_display(datum) [Thu Nov 21 17:09:40.192990 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] File "/opt/lib/python/site-packages/horizon/tables/base.py", line 1835, in get_object_display [Thu Nov 21 17:09:40.192992 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] display_key = self.get_object_display_key(datum) [Thu Nov 21 17:09:40.192994 2024] [wsgi:error] [pid 9:tid 124] [remote 208.80.155.117:56596] TypeError: get_object_display_key() takes 0 positional arguments but 2 were given
dcaro triaged T380511: [horizon] failing to show the proxies tab for some projects as High priority.
dcaro changed the status of T380489: CephSlowOps Ceph cluster in eqiad has 253 slow ops from Open to In Progress.
dcaro changed the status of T380239: CephSlowOps Ceph cluster in eqiad has 1 slow ops from Open to In Progress.
dcaro added a comment to T380489: CephSlowOps Ceph cluster in eqiad has 253 slow ops.
Might be related to T380239
dcaro added a comment to T380239: CephSlowOps Ceph cluster in eqiad has 1 slow ops.
From the switch graphs, something made it blip:
dcaro added a comment to T380489: CephSlowOps Ceph cluster in eqiad has 253 slow ops.
dcaro added a comment to T380489: CephSlowOps Ceph cluster in eqiad has 253 slow ops.
dcaro added a comment to T380489: CephSlowOps Ceph cluster in eqiad has 253 slow ops.
Having slow heartbeats:
Slow OSD heartbeats on back (longest 6757.204ms) Slow OSD heartbeats on front (longest 6091.493ms)
Nov 20 2024
Nov 20 2024
dcaro closed T348633: [api-gateway] add alert for uptime, a subtask of T348634: ceph slow ops 2023-10-11, as Resolved.
dcaro reopened T348633: [api-gateway] add alert for uptime as "In Progress".
dcaro moved T348633: [api-gateway] add alert for uptime from In Review to In Progress on the Toolforge (Toolforge iteration 16) board.
dcaro reopened T348633: [api-gateway] add alert for uptime, a subtask of T348634: ceph slow ops 2023-10-11, as In Progress.
dcaro closed T348633: [api-gateway] add alert for uptime, a subtask of T348634: ceph slow ops 2023-10-11, as Resolved.