**Describe the bug** When removing lots of components using `DELETE /api/components/(string:project)/(string:component)/`. Deadlocks may raise and not all components will get removed. A few hours later after DELETE requests, logs stopped with no more CPU consumption. But only several of them were removed, leaving logs below. **To Reproduce** This happened after #4398 After removing components failed in the addon script, I tried to remove them using API. I list all the unwanted components and sent a bunch of DELETE requests to the server. **Expected behavior** All unwanted components are removed eventually. **Screenshots** **Server configuration and status** **Exception traceback** ``` weblate_1 | celery-main stderr | [2020-08-29 04:26:52,377: INFO/MainProcess] Received task: weblate.trans.tasks.component_removal[96b2ff92-8ef2-4da5-90be-0dc156759223] weblate_1 | celery-main stderr | ERROR Failure while executing task: OperationalError: deadlock detected weblate_1 | celery-main stderr | DETAIL: Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836. weblate_1 | celery-main stderr | Process 24836 waits for ShareLock on transaction 105389; blocked by process 24833. weblate_1 | celery-main stderr | HINT: See server log for query details. weblate_1 | celery-main stderr | CONTEXT: while updating tuple (18,112) in relation "trans_alert" weblate_1 | celery-main stderr | weblate_1 | celery-main stderr | [2020-08-29 04:27:05,722: ERROR/ForkPoolWorker-2269] Failure while executing task: OperationalError: deadlock detected weblate_1 | celery-main stderr | DETAIL: Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836. weblate_1 | celery-main stderr | Process 24836 waits for ShareLock on transaction 105389; blocked by process 24833. weblate_1 | celery-main stderr | HINT: See server log for query details. weblate_1 | celery-main stderr | CONTEXT: while updating tuple (18,112) in relation "trans_alert" weblate_1 | celery-main stderr | weblate_1 | celery-main stderr | ERROR Failure while executing task: OperationalError: {'signal': , 'sender': <@task: weblate.trans.tasks.component_removal of weblate at 0x7f9d9feae8d0>, 'task_id': 'be2f6bb8-94ab-4f4e-9e66-a76b7be38c12', 'args': [5821, 3], 'kwargs': {}, 'traceback': , 'einfo': } weblate_1 | celery-main stderr | [2020-08-29 04:27:05,722: ERROR/ForkPoolWorker-2269] Failure while executing task: OperationalError: {'signal': , 'sender': <@task: weblate.trans.tasks.component_removal of weblate at 0x7f9d9feae8d0>, 'task_id': 'be2f6bb8-94ab-4f4e-9e66-a76b7be38c12', 'args': [5821, 3], 'kwargs': {}, 'traceback': , 'einfo': } weblate_1 | celery-main stderr | ERROR Failure while executing task weblate_1 | celery-main stderr | Traceback (most recent call last): weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute weblate_1 | celery-main stderr | return self.cursor.execute(sql, params) weblate_1 | celery-main stderr | psycopg2.errors.DeadlockDetected: deadlock detected weblate_1 | celery-main stderr | DETAIL: Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836. weblate_1 | celery-main stderr | Process 24836 waits for ShareLock on transaction 105389; blocked by process 24833. weblate_1 | celery-main stderr | HINT: See server log for query details. weblate_1 | celery-main stderr | CONTEXT: while updating tuple (18,112) in relation "trans_alert" weblate_1 | celery-main stderr | The above exception was the direct cause of the following exception: weblate_1 | celery-main stderr | weblate_1 | celery-main stderr | Traceback (most recent call last): weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 412, in trace_task weblate_1 | celery-main stderr | R = retval = fun(*args, **kwargs) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 704, in __protected_call__ weblate_1 | celery-main stderr | return self.run(*args, **kwargs) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/integrations/celery.py", line 171, in _inner weblate_1 | celery-main stderr | reraise(*exc_info) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/_compat.py", line 57, in reraise weblate_1 | celery-main stderr | raise value weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/integrations/celery.py", line 166, in _inner weblate_1 | celery-main stderr | return f(*args, **kwargs) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/tasks.py", line 299, in component_removal weblate_1 | celery-main stderr | obj.delete() weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/utils/db.py", line 127, in delete weblate_1 | celery-main stderr | return collector.delete() weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/deletion.py", line 436, in delete weblate_1 | celery-main stderr | sender=model, instance=obj, using=self.using weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/dispatch/dispatcher.py", line 179, in send weblate_1 | celery-main stderr | for receiver in self._live_receivers(sender) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/dispatch/dispatcher.py", line 179, in weblate_1 | celery-main stderr | for receiver in self._live_receivers(sender) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/utils/decorators.py", line 30, in wrapper weblate_1 | celery-main stderr | signal_handler(*args, **kwargs) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/__init__.py", line 223, in post_delete_linked weblate_1 | celery-main stderr | instance.linked_component.update_alerts() weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 2313, in update_alerts weblate_1 | celery-main stderr | self.add_alert("DuplicateFilemask", duplicates=duplicates) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1629, in add_alert weblate_1 | celery-main stderr | component.add_alert(alert, **details) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1626, in add_alert weblate_1 | celery-main stderr | obj.save() weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/alert.py", line 61, in save weblate_1 | celery-main stderr | super().save(*args, **kwargs) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 751, in save weblate_1 | celery-main stderr | force_update=force_update, update_fields=update_fields) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 789, in save_base weblate_1 | celery-main stderr | force_update, using, update_fields, weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 870, in _save_table weblate_1 | celery-main stderr | forced_update) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 923, in _do_update weblate_1 | celery-main stderr | return filtered._update(values) > 0 weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 803, in _update weblate_1 | celery-main stderr | return query.get_compiler(self.db).execute_sql(CURSOR) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1515, in execute_sql weblate_1 | celery-main stderr | cursor = super().execute_sql(result_type) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1154, in execute_sql weblate_1 | celery-main stderr | cursor.execute(sql, params) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/integrations/django/__init__.py", line 489, in execute weblate_1 | celery-main stderr | return real_execute(self, sql, params) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 66, in execute weblate_1 | celery-main stderr | return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers weblate_1 | celery-main stderr | return executor(sql, params, many, context) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute weblate_1 | celery-main stderr | return self.cursor.execute(sql, params) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/utils.py", line 90, in __exit__ weblate_1 | celery-main stderr | raise dj_exc_value.with_traceback(traceback) from exc_value weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute weblate_1 | celery-main stderr | return self.cursor.execute(sql, params) weblate_1 | celery-main stderr | django.db.utils.OperationalError: deadlock detected weblate_1 | celery-main stderr | DETAIL: Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836. weblate_1 | celery-main stderr | Process 24836 waits for ShareLock on transaction 105389; blocked by process 24833. weblate_1 | celery-main stderr | HINT: See server log for query details. weblate_1 | celery-main stderr | CONTEXT: while updating tuple (18,112) in relation "trans_alert" weblate_1 | celery-main stderr | weblate_1 | celery-main stderr | [2020-08-29 04:27:05,723: ERROR/ForkPoolWorker-2269] Failure while executing task weblate_1 | celery-main stderr | Traceback (most recent call last): weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute weblate_1 | celery-main stderr | return self.cursor.execute(sql, params) weblate_1 | celery-main stderr | psycopg2.errors.DeadlockDetected: deadlock detected weblate_1 | celery-main stderr | DETAIL: Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836. weblate_1 | celery-main stderr | Process 24836 waits for ShareLock on transaction 105389; blocked by process 24833. weblate_1 | celery-main stderr | HINT: See server log for query details. weblate_1 | celery-main stderr | CONTEXT: while updating tuple (18,112) in relation "trans_alert" weblate_1 | celery-main stderr | weblate_1 | celery-main stderr | weblate_1 | celery-main stderr | The above exception was the direct cause of the following exception: weblate_1 | celery-main stderr | weblate_1 | celery-main stderr | Traceback (most recent call last): weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 412, in trace_task weblate_1 | celery-main stderr | R = retval = fun(*args, **kwargs) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 704, in __protected_call__ weblate_1 | celery-main stderr | return self.run(*args, **kwargs) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/integrations/celery.py", line 171, in _inner weblate_1 | celery-main stderr | reraise(*exc_info) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/_compat.py", line 57, in reraise weblate_1 | celery-main stderr | raise value weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/integrations/celery.py", line 166, in _inner weblate_1 | celery-main stderr | return f(*args, **kwargs) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/tasks.py", line 299, in component_removal weblate_1 | celery-main stderr | obj.delete() weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/utils/db.py", line 127, in delete weblate_1 | celery-main stderr | return collector.delete() weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/deletion.py", line 436, in delete weblate_1 | celery-main stderr | sender=model, instance=obj, using=self.using weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/dispatch/dispatcher.py", line 179, in send weblate_1 | celery-main stderr | for receiver in self._live_receivers(sender) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/dispatch/dispatcher.py", line 179, in weblate_1 | celery-main stderr | for receiver in self._live_receivers(sender) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/utils/decorators.py", line 30, in wrapper weblate_1 | celery-main stderr | signal_handler(*args, **kwargs) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/__init__.py", line 223, in post_delete_linked weblate_1 | celery-main stderr | instance.linked_component.update_alerts() weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 2313, in update_alerts weblate_1 | celery-main stderr | self.add_alert("DuplicateFilemask", duplicates=duplicates) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1629, in add_alert weblate_1 | celery-main stderr | component.add_alert(alert, **details) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/component.py", line 1626, in add_alert weblate_1 | celery-main stderr | obj.save() weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/weblate/trans/models/alert.py", line 61, in save weblate_1 | celery-main stderr | super().save(*args, **kwargs) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 751, in save weblate_1 | celery-main stderr | force_update=force_update, update_fields=update_fields) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 789, in save_base weblate_1 | celery-main stderr | force_update, using, update_fields, weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 870, in _save_table weblate_1 | celery-main stderr | forced_update) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/base.py", line 923, in _do_update weblate_1 | celery-main stderr | return filtered._update(values) > 0 weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/query.py", line 803, in _update weblate_1 | celery-main stderr | return query.get_compiler(self.db).execute_sql(CURSOR) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1515, in execute_sql weblate_1 | celery-main stderr | cursor = super().execute_sql(result_type) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/models/sql/compiler.py", line 1154, in execute_sql weblate_1 | celery-main stderr | cursor.execute(sql, params) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/sentry_sdk/integrations/django/__init__.py", line 489, in execute weblate_1 | celery-main stderr | return real_execute(self, sql, params) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 66, in execute weblate_1 | celery-main stderr | return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers weblate_1 | celery-main stderr | return executor(sql, params, many, context) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute weblate_1 | celery-main stderr | return self.cursor.execute(sql, params) weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/utils.py", line 90, in __exit__ weblate_1 | celery-main stderr | raise dj_exc_value.with_traceback(traceback) from exc_value weblate_1 | celery-main stderr | File "/usr/local/lib/python3.7/dist-packages/django/db/backends/utils.py", line 84, in _execute weblate_1 | celery-main stderr | return self.cursor.execute(sql, params) weblate_1 | celery-main stderr | django.db.utils.OperationalError: deadlock detected weblate_1 | celery-main stderr | DETAIL: Process 24833 waits for ShareLock on transaction 105391; blocked by process 24836. weblate_1 | celery-main stderr | Process 24836 waits for ShareLock on transaction 105389; blocked by process 24833. weblate_1 | celery-main stderr | HINT: See server log for query details. weblate_1 | celery-main stderr | CONTEXT: while updating tuple (18,112) in relation "trans_alert" weblate_1 | celery-main stderr | weblate_1 | celery-notify stderr | [2020-08-29 04:27:06,718: INFO/MainProcess] Received task: weblate.accounts.tasks.notify_change[1b04ef9b-e548-4b08-9dee-04cd869c86a9] weblate_1 | celery-main stderr | [2020-08-29 04:27:08,421: INFO/MainProcess] Received task: weblate.trans.tasks.component_removal[5527a8f2-4af9-45b7-bae7-37ba4d32b7c9] weblate_1 | celery-main stderr | ERROR Failure while executing task: OperationalError: deadlock detected weblate_1 | celery-main stderr | DETAIL: Process 24836 waits for ShareLock on transaction 105393; blocked by process 24839. weblate_1 | celery-main stderr | Process 24839 waits for ShareLock on transaction 105391; blocked by process 24836. weblate_1 | celery-main stderr | HINT: See server log for query details. weblate_1 | celery-main stderr | CONTEXT: while updating tuple (18,113) in relation "trans_alert" weblate_1 | celery-main stderr | weblate_1 | celery-main stderr | [2020-08-29 04:27:24,429: ERROR/ForkPoolWorker-2270] Failure while executing task: OperationalError: deadlock detected weblate_1 | celery-main stderr | DETAIL: Process 24836 waits for ShareLock on transaction 105393; blocked by process 24839. weblate_1 | celery-main stderr | Process 24839 waits for ShareLock on transaction 105391; blocked by process 24836. weblate_1 | celery-main stderr | HINT: See server log for query details. weblate_1 | celery-main stderr | CONTEXT: while updating tuple (18,113) in relation "trans_alert" weblate_1 | celery-main stderr | ``` **Additional context** Since there is a deadlock, it might be done one by one. I rewrote the script, making it always checking if the last component is removed before sending DELETE request for the next component. This scripts worked and all unwanted components are removed.

Can not remove lots of components by sending requests to API in a short time period due to deadlock #4415

Company

Community

Support

Connect