Page MenuHomePhabricator

toolforge: maintain-kubeusers crashes if LDAP server terminates session
Closed, DuplicatePublic

Description

I have seen the maintain-kubeusers pod get restarted because this traceback:

2024-06-26T05:24:47.726416427Z starting a run
2024-06-26T05:24:54.010266131Z invalid account name 'ru_monuments', skipping
2024-06-26T05:24:54.045146435Z invalid account name 'wdq_checker', skipping
2024-06-26T05:25:08.881721420Z Traceback (most recent call last):
2024-06-26T05:25:08.881763443Z   File "/app/maintain_kubeusers_cli.py", line 6, in <module>
2024-06-26T05:25:08.881831408Z     runpy.run_module("maintain_kubeusers", run_name="__main__")
2024-06-26T05:25:08.882014324Z   File "<frozen runpy>", line 229, in run_module
2024-06-26T05:25:08.882026160Z   File "<frozen runpy>", line 88, in _run_code
2024-06-26T05:25:08.882034187Z   File "/app/maintain_kubeusers/__main__.py", line 6, in <module>
2024-06-26T05:25:08.882097543Z     main()
2024-06-26T05:25:08.882149572Z   File "/app/maintain_kubeusers/cli.py", line 170, in main
2024-06-26T05:25:08.882249903Z     do_run(
2024-06-26T05:25:08.882261985Z   File "<decorator-gen-1>", line 2, in do_run
2024-06-26T05:25:08.882290411Z   File "/opt/lib/python/site-packages/prometheus_client/context_managers.py", line 80, in wrapped
2024-06-26T05:25:08.882360931Z     return func(*args, **kwargs)
2024-06-26T05:25:08.882413130Z            ^^^^^^^^^^^^^^^^^^^^^
2024-06-26T05:25:08.882422032Z   File "/app/maintain_kubeusers/cli.py", line 79, in do_run
2024-06-26T05:25:08.882509629Z     admins = get_admins_from_ldap(
2024-06-26T05:25:08.882520006Z              ^^^^^^^^^^^^^^^^^^^^^
2024-06-26T05:25:08.882527864Z   File "/app/maintain_kubeusers/utils.py", line 98, in get_admins_from_ldap
2024-06-26T05:25:08.882594703Z     for entry in entries:
2024-06-26T05:25:08.882603861Z   File "/opt/lib/python/site-packages/ldap3/extend/standard/PagedSearch.py", line 56, in paged_search_generator
2024-06-26T05:25:08.882702685Z     result = connection.search(search_base,
2024-06-26T05:25:08.882735513Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-26T05:25:08.882767475Z   File "/opt/lib/python/site-packages/ldap3/core/connection.py", line 853, in search
2024-06-26T05:25:08.882944206Z     response = self.post_send_search(self.send('searchRequest', request, controls))
2024-06-26T05:25:08.883009808Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-26T05:25:08.883019084Z   File "/opt/lib/python/site-packages/ldap3/strategy/sync.py", line 178, in post_send_search
2024-06-26T05:25:08.883122482Z     responses, result = self.get_response(message_id)
2024-06-26T05:25:08.883154210Z                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-26T05:25:08.883162878Z   File "/opt/lib/python/site-packages/ldap3/strategy/base.py", line 370, in get_response
2024-06-26T05:25:08.883337263Z     raise LDAPSessionTerminatedByServerError(self.connection.last_error)
2024-06-26T05:25:08.883357090Z ldap3.core.exceptions.LDAPSessionTerminatedByServerError: session terminated by server

We could explore if gracefully handling this LDAP server error would be better.

  NODES
admin 3
Note 1
USERS 13