Docspace backup hung

ivan · 18 February 2025 06:22

Hi, just updated to docspace 3.0.1 (it went well), but when I tried to do an on demand backup to my Backup Room as I had been doing the backup seems to be hung.

Running.
table /onlyoffice-router\tonlyoffice/docspace-router:3.0.1.1
table /onlyoffice-login\tonlyoffice/docspace-login:3.0.1.1
table /onlyoffice-doceditor\tonlyoffice/docspace-doceditor:3.0.1.1
table /onlyoffice-backup\tonlyoffice/docspace-backup:3.0.1.1
table /onlyoffice-studio\tonlyoffice/docspace-studio:3.0.1.1
table /onlyoffice-backup-background-tasks\tonlyoffice/docspace-backup-background:3.0.1.1
table /onlyoffice-files\tonlyoffice/docspace-files:3.0.1.1
table /onlyoffice-files-services\tonlyoffice/docspace-files-services:3.0.1.1
table /onlyoffice-people-server\tonlyoffice/docspace-people-server:3.0.1.1
table /onlyoffice-ssoauth\tonlyoffice/docspace-ssoauth:3.0.1.1
table /onlyoffice-clear-events\tonlyoffice/docspace-clear-events:3.0.1.1
table /onlyoffice-notify\tonlyoffice/docspace-notify:3.0.1.1
table /onlyoffice-api-system\tonlyoffice/docspace-api-system:3.0.1.1
table /onlyoffice-api\tonlyoffice/docspace-api:3.0.1.1
table /onlyoffice-studio-notify\tonlyoffice/docspace-studio-notify:3.0.1.1
table /onlyoffice-socket\tonlyoffice/docspace-socket:3.0.1.1
table /onlyoffice-proxy\tnginx:latest
table /onlyoffice-healthchecks\tonlyoffice/docspace-healthchecks:3.0.1.1
table /onlyoffice-identity-authorization\tonlyoffice/docspace-identity-authorization:3.0.1.1
table /onlyoffice-identity-api\tonlyoffice/docspace-identity-api:3.0.1.1
table /onlyoffice-document-server\tonlyoffice/documentserver:8.3.0.1
table /onlyoffice-rabbitmq\trabbitmq:3.13.6
table /onlyoffice-redis\tredis:7
table /onlyoffice-mysql-server\tmysql:8.3.0
table /onlyoffice-opensearch-dashboards\topensearchproject/opensearch-dashboards:2.11.1
table /onlyoffice-opensearch\tonlyoffice/opensearch:2.11.1
table /onlyoffice-fluent-bit\tfluent/fluent-bit:3.0.2

And I see errors here /var/lib/docker/volumes/onlyoffice_log_data/_data/ASC.Data.Backup.BackgroundTasks.log

at ASC.EventBus.RabbitMQ.DefaultRabbitMQPersistentConnection.<TryConnectAsync>b__10_2() in /app/onlyoffice/src/server/common/ASC.EventBus.RabbitMQ/DefaultRabbitMQPersistentConnection.cs:line 105
   at Polly.AsyncPolicy.<>c__DisplayClass44_0.<<ImplementationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, CancellationToken cancellationToken, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
2025-02-18 06:20:23,320|WARN|[22]|ASC.EventBus.RabbitMQ.DefaultRabbitMQPersistentConnection - RabbitMQ Client could not connect after 8s|RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable
 ---> RabbitMQ.Client.Exceptions.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
   at RabbitMQ.Client.Framing.Connection.StartAndTuneAsync(CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.Connection.OpenAsync(CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.Connection.OpenAsync(CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.AutorecoveringConnection.CreateAsync(ConnectionConfig config, IEndpointResolver endpoints, CancellationToken cancellationToken)
   at RabbitMQ.Client.ConnectionFactory.CreateConnectionAsync(IEndpointResolver endpointResolver, String clientProvidedName, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at RabbitMQ.Client.ConnectionFactory.CreateConnectionAsync(IEndpointResolver endpointResolver, String clientProvidedName, CancellationToken cancellationToken)
   at ASC.EventBus.RabbitMQ.DefaultRabbitMQPersistentConnection.<TryConnectAsync>b__10_2() in /app/onlyoffice/src/server/common/ASC.EventBus.RabbitMQ/DefaultRabbitMQPersistentConnection.cs:line 105
   at Polly.AsyncPolicy.<>c__DisplayClass44_0.<<ImplementationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, CancellationToken cancellationToken, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
2025-02-18 06:20:31,330|WARN|[10]|ASC.EventBus.RabbitMQ.DefaultRabbitMQPersistentConnection - RabbitMQ Client could not connect after 16s|RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable
 ---> RabbitMQ.Client.Exceptions.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.
   at RabbitMQ.Client.Framing.Connection.StartAndTuneAsync(CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.Connection.OpenAsync(CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.Connection.OpenAsync(CancellationToken cancellationToken)
   at RabbitMQ.Client.Framing.AutorecoveringConnection.CreateAsync(ConnectionConfig config, IEndpointResolver endpoints, CancellationToken cancellationToken)
   at RabbitMQ.Client.ConnectionFactory.CreateConnectionAsync(IEndpointResolver endpointResolver, String clientProvidedName, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at RabbitMQ.Client.ConnectionFactory.CreateConnectionAsync(IEndpointResolver endpointResolver, String clientProvidedName, CancellationToken cancellationToken)
   at ASC.EventBus.RabbitMQ.DefaultRabbitMQPersistentConnection.<TryConnectAsync>b__10_2() in /app/onlyoffice/src/server/common/ASC.EventBus.RabbitMQ/DefaultRabbitMQPersistentConnection.cs:line 105
   at Polly.AsyncPolicy.<>c__DisplayClass44_0.<<ImplementationAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, CancellationToken cancellationToken, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)

My screen looks like (lots of greyed out buttons) and no “progress spinner” at bottom right

Any help appreciated.

Can i simple abort this old manual backup?

Thanks
Ivan

ivan · 18 February 2025 17:05

This is more serious and maybe related.

These TWO identity containers keep restarting, never going healthy.

onlyoffice/docspace-identity-authorization:3.0.1.1
onlyoffice/docspace-identity-api:3.0.1.1

The CPUs are pegged at 400%

The service is now unusable.

Any help needed, it was fine before 3.01

Error logs look like

{"date":"2025-02-18T17:02:11.590223129Z","@version":"1","message":"Application run failed","sourceContext":"org.springframework.boot.SpringApplication","threadId":"main","level":"ERROR","stack_trace":"c.r.c.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.\n\tat c.r.c.impl.AMQConnection.start(AMQConnection.java:395)\n\tat c.r.c.ConnectionFactory.newConnection(ConnectionFactory.java:1251)\n\tat c.r.c.ConnectionFactory.newConnection(ConnectionFactory.java:1198)\n\tat o.s.a.r.c.AbstractConnectionFactory.connectAddresses(AbstractConnectionFactory.java:668)\n\t... 44 frames truncated (including 42 common frames)\nWrapped by: o.s.a.AmqpAuthenticationException: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.\n\tat o.s.a.r.s.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:64)\n\tat o.s.a.r.c.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:622)\n\tat o.s.a.r.c.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:726)\n\tat o.s.a.r.c.ConnectionFactoryUtils.createConnection(ConnectionFactoryUtils.java:257)\n\t... 40 frames truncated (including 25 common frames)\nWrapped by: o.s.b.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.rabbit.core.RabbitAdmin]: Factory method 'rabbitAdmin' threw exception with message: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.\n\tat o.s.b.f.s.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:177)\n\tat o.s.b.f.s.ConstructorResolver.instantiate(ConstructorResolver.java:644)\n\t... 24 common frames omitted\nWrapped by: o.s.b.f.BeanCreationException: Error creating bean with name 'rabbitAdmin' defined in class path resource [com/asc/common/messaging/configuration/RabbitMQConfiguration.class]: Failed to instantiate [org.springframework.amqp.rabbit.core.RabbitAdmin]: Factory method 'rabbitAdmin' threw exception with message: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.\n\tat o.s.b.f.s.ConstructorResolver.instantiate(ConstructorResolver.java:648)\n\tat o.s.b.f.s.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636)\n\tat o.s.b.f.s.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1337)\n\tat o.s.b.f.s.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1167)\n\t... 21 frames truncated\n","applicationContext":"ASC.Identity.Authorization.API","instanceId":"58b2ff67a8b4"}
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::                (v3.3.0)

{"date":"2025-02-18T17:02:44.089203446Z","@version":"1","message":"Application run failed","sourceContext":"org.springframework.boot.SpringApplication","threadId":"main","level":"ERROR","stack_trace":"c.r.c.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.\n\tat c.r.c.impl.AMQConnection.start(AMQConnection.java:395)\n\tat c.r.c.ConnectionFactory.newConnection(ConnectionFactory.java:1251)\n\tat c.r.c.ConnectionFactory.newConnection(ConnectionFactory.java:1198)\n\tat o.s.a.r.c.AbstractConnectionFactory.connectAddresses(AbstractConnectionFactory.java:668)\n\t... 44 frames truncated (including 42 common frames)\nWrapped by: o.s.a.AmqpAuthenticationException: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.\n\tat o.s.a.r.s.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:64)\n\tat o.s.a.r.c.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:622)\n\tat o.s.a.r.c.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:726)\n\tat o.s.a.r.c.ConnectionFactoryUtils.createConnection(ConnectionFactoryUtils.java:257)\n\t... 40 frames truncated (including 25 common frames)\nWrapped by: o.s.b.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.rabbit.core.RabbitAdmin]: Factory method 'rabbitAdmin' threw exception with message: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.\n\tat o.s.b.f.s.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:177)\n\tat o.s.b.f.s.ConstructorResolver.instantiate(ConstructorResolver.java:644)\n\t... 24 common frames omitted\nWrapped by: o.s.b.f.BeanCreationException: Error creating bean with name 'rabbitAdmin' defined in class path resource [com/asc/common/messaging/configuration/RabbitMQConfiguration.class]: Failed to instantiate [org.springframework.amqp.rabbit.core.RabbitAdmin]: Factory method 'rabbitAdmin' threw exception with message: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.\n\tat o.s.b.f.s.ConstructorResolver.instantiate(ConstructorResolver.java:648)\n\tat o.s.b.f.s.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636)\n\tat o.s.b.f.s.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1337)\n\tat o.s.b.f.s.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1167)\n\t... 21 frames truncated\n","applicationContext":"ASC.Identity.Authorization.API","instanceId":"58b2ff67a8b4"}
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::                (v3.3.0)

{"date":"2025-02-18T17:03:17.973078269Z","@version":"1","message":"Application run failed","sourceContext":"org.springframework.boot.SpringApplication","threadId":"main","level":"ERROR","stack_trace":"c.r.c.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.\n\tat c.r.c.impl.AMQConnection.start(AMQConnection.java:395)\n\tat c.r.c.ConnectionFactory.newConnection(ConnectionFactory.java:1251)\n\tat c.r.c.ConnectionFactory.newConnection(ConnectionFactory.java:1198)\n\tat o.s.a.r.c.AbstractConnectionFactory.connectAddresses(AbstractConnectionFactory.java:668)\n\t... 44 frames truncated (including 42 common frames)\nWrapped by: o.s.a.AmqpAuthenticationException: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.\n\tat o.s.a.r.s.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:64)\n\tat o.s.a.r.c.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:622)\n\tat o.s.a.r.c.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:726)\n\tat o.s.a.r.c.ConnectionFactoryUtils.createConnection(ConnectionFactoryUtils.java:257)\n\t... 40 frames truncated (including 25 common frames)\nWrapped by: o.s.b.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.rabbit.core.RabbitAdmin]: Factory method 'rabbitAdmin' threw exception with message: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.\n\tat o.s.b.f.s.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:177)\n\tat o.s.b.f.s.ConstructorResolver.instantiate(ConstructorResolver.java:644)\n\t... 24 common frames omitted\nWrapped by: o.s.b.f.BeanCreationException: Error creating bean with name 'rabbitAdmin' defined in class path resource [com/asc/common/messaging/configuration/RabbitMQConfiguration.class]: Failed to instantiate [org.springframework.amqp.rabbit.core.RabbitAdmin]: Factory method 'rabbitAdmin' threw exception with message: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.\n\tat o.s.b.f.s.ConstructorResolver.instantiate(ConstructorResolver.java:648)\n\tat o.s.b.f.s.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636)\n\tat o.s.b.f.s.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1337)\n\tat o.s.b.f.s.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1167)\n\t... 21 frames truncated\n","applicationContext":"ASC.Identity.Authorization.API","instanceId":"58b2ff67a8b4"}
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::                (v3.3.0)

type or paste code here

ivan · 18 February 2025 17:49

Rabbit is broken

2025-02-18 15:51:20.876149+00:00 [error] <0.92992.0> Error on AMQP connection <0.92992.0> (172.18.0.17:47936 → 172.18.0.18:5672, state: starting):
2025-02-18 15:51:20.876149+00:00 [error] <0.92992.0> PLAIN login refused: user ‘guest’ - invalid credentials
2025-02-18 15:51:20.877473+00:00 [info] <0.92992.0> closing AMQP connection <0.92992.0> (172.18.0.17:47936 → 172.18.0.18:5672 - BackupBackgroundTasks)
2025-02-18 15:51:25.395771+00:00 [info] <0.92997.0> accepting AMQP connection <0.92997.0> (172.18.0.25:36984 → 172.18.0.18:5672)
2025-02-18 15:51:25.399415+00:00 [info] <0.92997.0> connection <0.92997.0> (172.18.0.25:36984 → 172.18.0.18:5672) has a client-provided name: Notify
2025-02-18 15:51:25.399710+00:00 [error] <0.92997.0> Error on AMQP connection <0.92997.0> (172.18.0.25:36984 → 172.18.0.18:5672, state: starting):
2025-02-18 15:51:25.399710+00:00 [error] <0.92997.0> PLAIN login refused: user ‘guest’ - invalid credentials
2025-02-18 15:51:25.401183+00:00 [info] <0.92997.0> closing AMQP connection <0.92997.0> (172.18.0.25:36984 → 172.18.0.18:5672 - Notify)
2025-02-18 15:51:26.010918+00:00 [info] <0.93002.0> accepting AMQP connection <0.93002.0> (172.18.0.7:46994 → 172.18.0.18:5672)
2025-02-18 15:51:26.014724+00:00 [info] <0.93002.0> connection <0.93002.0> (172.18.0.7:46994 → 172.18.0.18:5672) has a client^C
2025-02-18 15:52:04.170488+00:00 [error] <0.93073.0> PLAIN login refused: user ‘guest’ - invalid credentials
2025-02-18 15:52:04.176311+00:00 [info] <0.93073.0> closing AMQP connection <0.93073.0> (172.18.0.14:40150 → 172.18.0.18:5672 - FilesService)
2025-02-18 15:52:04.381064+00:00 [info] <0.93078.0> accepting AMQP connection <0.93078.0> (172.18.0.20:40334 → 172.18.0.18:5672)
2025-02-18 15:52:04.453374+00:00 [info] <0.93078.0> connection <0.93078.0> (172.18.0.20:40334 → 172.18.0.18:5672) has a client-provided name: rabbitConnectionFactory#23b31c6c:0
2025-02-18 15:52:04.453744+00:00 [error] <0.93078.0> Error on AMQP connection <0.93078.0> (172.18.0.20:40334 → 172.18.0.18:5672, state: starting):
2025-02-18 15:52:04.453744+00:00 [error] <0.93078.0> PLAIN login refused: user ‘guest’ - invalid credentials
2025-02-18 15:52:04.455218+00:00 [info] <0.93078.0> closing AMQP connection <0.93078.0> (172.18.0.20:40334 → 172.18.0.18:5672 - rabbitConnectionFactory#23b31c6c:0)
2025-02-18 15:52:08.189560+00:00 [info] <0.93083.0> accepting AMQP connection <0.93083.0> (172.18.0.14:48004 → 172.18.0.18:5672)

Alexandre · 19 February 2025 09:57

Dear @ivan
How exactly did you perform the upgrade process? I’ve checked docker installation on my Ubuntu 24.04 server, all things are OK, the mentioned containers are running and the backup creation is OK.
According to the provided logs, it seems that the issue is related to RabbitMQ. Could you please provide us with entire DocSpace logs folder (/var/lib/docker/volumes/onlyoffice_log_data/_data/
)?
Additionally, please go RabbitMQ container (docker exec -it ID bash) and check its status with rabbitmqctl status command. Please show us the output result.

ivan · 19 February 2025 15:16

H Alexandre.

As root:
wget http://download.onlyoffice.com/docspace/docspace-install.sh
bash docspace-install.sh -u true

(there were no on screen errors)

rabbit.log (2.7 KB)

logs.docx (6.5 MB)

Please rename logs.docx to logs.tar.gz and gunzip to the contents of full logfolder

many thanks
Ivan