Unable to get Community Server 11.5.2.1542 running

Workspace
1

Hi all,

I have OnlyOffice Workspace Community Edition running, with Community Server version 11.5.2.1542 and Control Panel v 3.0.0.381 (no Mail Server or Docs) via Docker on a Debian Buster (10) host.

I have been doing regular backups via the built-in backup-and-restore function of Control Panel. Unfortunately (entirely my fault) I waited quite long before testing a restore of the backups. Now I want to create a test environment to test a restore of the backups (and test an upgrade to the current version after that). Alas, I cannot by the life of me get it to run. (And a restore of the backup in an environment with the current version of all components fails as well.)

I install the docker containers via

sudo bash workspace-install.sh -cv 11.5.2.1542 -cpv 3.0.0.381 -mk MACHINE-KEY-OF-PRODUCTION-INSTANCE

on a separate Debian Buster host.

Result is a “500 Internal Server Error” response. It seems mono is constantly restarting inside the onlyoffice/communityserver container.

Tail of /app/onlyoffice/CommunityServer/logs/web.log on host:

2024-03-20 16:10:22,581 ERROR [5] ASC.Web - Warmup error System.Collections.Generic.KeyNotFoundException: The given key '3894' was not present in the dictionary.
  at System.Collections.Generic.Dictionary`2[TKey,TValue].get_Item (TKey key) [0x0001e] in <f2b3ab7dfff746f594d2ef5b16ec3c90>:0 
  at MySql.Data.MySqlClient.MySqlField.SetFieldEncoding () [0x00034] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.MySqlField.set_CharacterSetIndex (System.Int32 value) [0x00007] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.NativeDriver.GetColumnData (MySql.Data.MySqlClient.MySqlField field) [0x000ad] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.NativeDriver.GetColumnsData (MySql.Data.MySqlClient.MySqlField[] columns) [0x00004] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.Driver.GetColumns (System.Int32 count) [0x0001c] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.ResultSet.LoadColumns (System.Int32 numCols) [0x00000] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.ResultSet..ctor (MySql.Data.MySqlClient.Driver d, System.Int32 statementId, System.Int32 numCols) [0x0002a] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.Driver.NextResult (System.Int32 statementId, System.Boolean force) [0x00039] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.MySqlDataReader.NextResult () [0x000d5] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.MySqlDataReader.Close () [0x00039] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.MySqlDataReader.Dispose (System.Boolean disposing) [0x00003] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.MySqlDataReader.Dispose () [0x00000] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery () [0x0004b] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.Driver.Configure (MySql.Data.MySqlClient.MySqlConnection connection) [0x0017e] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at MySql.Data.MySqlClient.MySqlConnection.Open () [0x002c8] in <2fd3c7a64a4546cfbdfc4da289d6e0a8>:0 
  at ASC.Common.Data.AdoProxy.DbConnectionProxy.Open () [0x00012] in <aa72a3ad0a754725806c8a1beb2f98a3>:0 
  at ASC.Common.Data.DbManager.OpenConnection () [0x00006] in <aa72a3ad0a754725806c8a1beb2f98a3>:0 
  at ASC.Common.Data.DbManager.get_Command () [0x0000e] in <aa72a3ad0a754725806c8a1beb2f98a3>:0 
  at ASC.Common.Data.DbManager.ExecuteList (ASC.Common.Data.Sql.ISqlInstruction sql) [0x00000] in <aa72a3ad0a754725806c8a1beb2f98a3>:0 
  at ASC.Common.Data.DbManagerProxy.ExecuteList (ASC.Common.Data.Sql.ISqlInstruction sql) [0x00006] in <aa72a3ad0a754725806c8a1beb2f98a3>:0 
  at ASC.Core.Data.DbBaseService+<>c__DisplayClass8_0.<ExecList>b__0 (ASC.Common.Data.IDbManager db) [0x00000] in <6c2b7679edd64973b811b6d5b86c5e26>:0 
  at ASC.Core.Data.DbBaseService.Execute[T] (System.Func`2[T,TResult] action) [0x00007] in <6c2b7679edd64973b811b6d5b86c5e26>:0 
  at ASC.Core.Data.DbBaseService.ExecList (ASC.Common.Data.Sql.ISqlInstruction sql) [0x0000d] in <6c2b7679edd64973b811b6d5b86c5e26>:0 
  at ASC.Core.Data.DbTenantService.GetTenants (ASC.Common.Data.Sql.Expressions.Exp where) [0x00007] in <6c2b7679edd64973b811b6d5b86c5e26>:0 
  at ASC.Core.Data.DbTenantService.GetTenants (System.DateTime from, System.Boolean active) [0x00048] in <6c2b7679edd64973b811b6d5b86c5e26>:0 
  at ASC.Core.Caching.CachedTenantService.GetTenants (System.DateTime from, System.Boolean active) [0x00000] in <6c2b7679edd64973b811b6d5b86c5e26>:0 
  at ASC.Core.TenantManager.GetTenants (System.Boolean active) [0x00000] in <6c2b7679edd64973b811b6d5b86c5e26>:0 
  at ASC.Web.Studio.Core.WarmUp..ctor () [0x0006b] in <3882346827c645f59d36da72df1ab97f>:0 
2024-03-20 16:10:22,581 ERROR [5] ASC - Start Warmup System.NullReferenceException: Object reference not set to an instance of an object
  at ASC.Web.Studio.Core.WarmUp.Start () [0x00033] in <3882346827c645f59d36da72df1ab97f>:0 
  at ASC.Web.Studio.Startup.Configure () [0x000f4] in <3882346827c645f59d36da72df1ab97f>:0

Within the communityserver container, a “service monoserve status” returns:

monoserve.service - FastCGI Mono server
   Loaded: loaded (/lib/systemd/system/monoserve.service; enabled; vendor preset: enabled)
   Active: deactivating (stop-sigterm) since Wed 2024-03-20 16:16:49 UTC; 499ms ago
  Process: 7099 ExecStartPre=/bin/bash -c test -e /var/run/onlyoffice || install -m 755 -o onlyoffice -g nginx -d /var/run/onlyoffice; rm -f /var/run/onlyoffice
/onlyoffice.socket /var/www/onlyoffice/WebStudio/mono_crash* /var/www/onlyoffice/WebStudio/core.*; (code=exited, status=0/SUCCESS)
 Main PID: 7101 (mono)
    Tasks: 62 (limit: 4915)
   CGroup: /docker/d4294e54f30c9dee94a45b24f1c216096b47fea5600b7a5d5602427b8d29ad8e/system.slice/monoserve.service
           └─7101 /usr/bin/mono /usr/lib/hyperfastcgi/4.0/HyperFastCgi.exe /config=/etc/hyperfastcgi/onlyoffice /logfile=/var/log/onlyoffice/onlyoffice.log /log
levels=Error

Mar 20 16:16:12 d4294e54f30c systemd[1]: Starting FastCGI Mono server...
Mar 20 16:16:12 d4294e54f30c systemd[1]: Started FastCGI Mono server.
Mar 20 16:16:13 d4294e54f30c hyperfastcgi4[7101]: WebConfigurationManager's LRUcache Size overriden to: 2000 (via MONO_ASPNET_WEBCONFIG_CACHESIZE)
Mar 20 16:16:49 d4294e54f30c systemd[1]: Stopping FastCGI Mono server...

I would highly appreciate any help. I do not want to risk the update procedure on the production instance without being able to test it first. Thanks!

2

Hello @martinsegur
Sorry for the late reply.
I see MySQL error entries in the provided log. Could you please double check that MySQL address is correct? Please go to/var/www/onlyoffice/Services/TeamLabSvc/TeamLabSvc.exe.config inside Community server container and check connectionStrings section. Additionally try to check mutual availability between the containers via wget or curl commands.
P.S. Side note, your Community server and Control Panel versions are outdated. We strongly recommend updating the portal timely.

3

Hi Alexandre,

thank you for offering your help!

I’ve checked /var/www/onlyoffice/Services/TeamLabSvc/TeamLabSvc.exe.config, the MySQL connection config seems fine (standard username and pw as per install script). I can log into the onlyoffice MySQL database from within the Community Server container using those credentials with

mysql -h onlyoffice-mysql-server  -u onlyoffice_user -p -P 3306 onlyoffice

Curl from MySQL container to Community Server container “works”, but returns 502 HTTP response:

bash-4.4# curl -o - -I onlyoffice-community-server
HTTP/1.1 502 Bad Gateway
Server: nginx/1.20.1
Date: Mon, 25 Mar 2024 10:59:43 GMT
Content-Type: text/html
Content-Length: 42241
Connection: keep-alive
ETag: "55312fb1-a501"

Curl from Community Server container to MySQL container returns this though:

root@d4294e54f30c:/var/www/onlyoffice/Services/TeamLabSvc# curl -o - -I onlyoffice-mysql-server:3306
curl: (8) Weird server reply

Best,
Martin.

P.S.: Oh I definitely want to update. But I can’t risk the update process without a working backup-restore :wink:

4

Hello @martinsegur

bash-4.4# curl -o - -I onlyoffice-community-server
HTTP/1.1 502 Bad Gateway

This is strange, my test new installation returns 302 Found in this scenario. Did you use clean freshly deployed server for this instance? Are there any proxy server or firewall in front of Workspace?

Additionally, please show us the output result of these commands:
df -h
free -h
docker logs onlyoffice-community-server

Also, please specify your exact OS and docker versions. One more thing, is it possible to provide us with entire Community server logs folder?

5

It was a clean server, but I’ve done some experiments there already, including trying a direct install, so without docker. I can do a wipe and set it all up again if necessary.

df -h

Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
udev             16G       0   16G    0% /dev
tmpfs           3,2G    9,4M  3,2G    1% /run
/dev/sda2       1,8T    121G  1,6T    7% /
tmpfs            16G     20K   16G    1% /dev/shm
tmpfs           5,0M    4,0K  5,0M    1% /run/lock
tmpfs            16G       0   16G    0% /sys/fs/cgroup
/dev/sda1       511M    5,8M  506M    2% /boot/efi
tmpfs           3,2G    4,0K  3,2G    1% /run/user/117
tmpfs           3,2G       0  3,2G    0% /run/user/0

free -h

              total        used        free      shared  buff/cache   available
Mem:           31Gi       435Mi        30Gi        20Mi       378Mi        30Gi
Swap:         975Mi          0B       975Mi

docker logs onlyoffice-community-server → see attached txt file.

cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

uname -r
4.19.0-26-amd64

docker -v
Docker version 25.0.4, build 1a576c5

I’ll get the docker logs and provide a download link in a moment.

6

Here is the download link for the folder /app/onlyoffice/CommunityServer/logs:

7

Hello @martinsegur
Thank you for the provided data, we are checking the situation. I will update this thread when we have something to share.

9

Hello @martinsegur
Probably we figured out the situation. There was a bug that was related to MySQL v.8.0.29 and it was fixed in the latest versions of Community server (v.12.0+), but it seems that you faced it on the outdated version (v.11.5). What is your MySQL version right now?
Could you please run a test and install Workspace via custom installation script on your second server from scratch? Please try to follow my steps:

Please let us know the result.

11

Dear @Alexandre,

Sorry for the late reply, only got around to trying this right now. A GIANT THANK YOU – it worked! I now have a working second environment, with a recovery of my latest backup from the production instance working perfectly. I will now test the update to the latest version.

Thank you so much for your help with this!

All the best and greetings from Austria,

Martin.

1 Like
12

We are glad that the situation has been resolved.
Please feel free to contact us if you face any issues.