Documentserver Healthcheck 502 Bad Gateway

Good day.
Recently I encountered a problem after updating an application installed on nextcloud.
When running the command:
sudo -u www php /usr/local/www/nextcloud/occ onlyoffice:documentserver --check
I get the error:
Error connection: Server error: GET http://my.url/healthcheck resulted in a 502 Bad Gateway.
Same error in the browser.
The links http://my.url/welcome/ and http://my.url/example/ work well.

$ sudo systemctl list-units | grep ds
ds-converter.service loaded active running Docs Converter
ds-docservice.service loaded active running Docs Docservice
ds-example.service loaded active running Docs Example
ds-metrics.service loaded active running Docs Metrics

Nginx configuration file:

map $http_upgrade $proxy_connection {
default upgrade;
“” close;
}

server {
access_log /var/log/nginx/docs.energy-yug.link.log.gz combined buffer=512k gzip=9;
add_header Strict-Transport-Security “max-age=31536000” always;
add_header X-Content-Type-Options “nosniff”;
http2 on;
include maintenance.conf;
include robots.conf;
listen 443 ssl;
proxy_cache local;
proxy_set_header Connection $proxy_connection;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
server_name my.url;

    location / {
            proxy_http_version 1.1;
            proxy_pass http://my.url;
    }
    location /healthcheck/ {
            proxy_http_version 1.1;
            proxy_pass http://my.url;
    }
    location ~ ^/(example|welcome)/ {
            proxy_http_version 1.1;
            proxy_pass http://my.url;
    }

}

Any ideas?

Hello @Sargatanas
Please provide us with details about the situation.

I encountered a problem after updating an application installed on nextcloud

Do you mean the connector app? Please let us know current version of this app and the version before you ran the update process.
Additionally, let us know your Document server and Nextcloud versions.
Do I understand it right, that you are using package installation of Document server? Please go to the host and run apt list --installed | grep onlyoffice (for CentOS, yum list installed | grep onlyoffice) command.
Do you use Nginx proxy server in front of Document server? Please compare your configuration file with our samples: Using ONLYOFFICE Docs behind the proxy - ONLYOFFICE
One more thing. Please reproduce the situation (go to the connector app and click Save buton) > make a screenshot of connector app page with error entry > please provide us with whole Document server logs folder. It’s located here: /var/log/onlyoffice/documentserver/

I have nexcloud, onlyoffice-documentserver and nginx installed on different machines.
No problems with nginx. The /welcome and /example pages open. I sent the nginx configuration above. I added the /healthcheck/ page, but it didn’t help.

Onlyoffice-documentserver is used exclusively as a server for online editing of documents from nextcloud.

Nextcloud - 26.0.7
Nextcloud app onlyoffice - 7.9.0
Onlyoffice onlyoffice-documentserver.x86_64 - 7.4.1-36.el7

Log files: https://cloud.energy-yug.link/s/PWDEwKyZsS9E459

Any ideas?

Regenerate a secret key and place it in the addon.

Before placing it, restart the machine where the documentserver is, then verify that this machine has access to the Internet. Since you must download the templates.

Go to the onlyoffice document server welcome and check that it is working.

You can show your local.json file
You find it in /etc/onlyoffice/documentserver/local.json

$ cat /etc/onlyoffice/documentserver/local.json
{
“services”: {
“CoAuthoring”: {
“sql”: {
“dbHost”: “localhost”,
“dbName”: “onlyoffice”,
“dbUser”: “postgres”,
“dbPass”: “”,
“type”: “postgres”,
“dbPort”: “5432”
},
“token”: {
“enable”: {
“request”: {
“inbox”: true,
“outbox”: true
},
“browser”: true
},
“inbox”: {
“header”: “Authorization”
},
“outbox”: {
“header”: “Authorization”
}
},
“secret”: {
“inbox”: {
“string”: “mysecret”
},
“outbox”: {
“string”: “mysecret”
},
“session”: {
“string”: “mysecret”
}
}
}
},
“queue”: {
“type”: “rabbitmq”
},
“rabbitmq”: {
“url”: “amqp://guest:guest@localhost”
},
“storage”: {
“fs”: {
“secretString”: “mysecretString”
}
}
}

Updated the secret key. It previously matched the settings in Nextcloud.
The “welcome” page is working.

I found the following lines in default.json:

    "services": {
            "CoAuthoring": {
                    "server": {
                            "port": 8000,
                            "workerpercpu": 1,
                            "mode": "development",
                            "limits_tempfile_upload": 104857600,
                            "limits_image_size": 26214400,
                            "limits_image_download_timeout": {
                                    "connectionAndInactivity": "2m",
                                    "wholeCycle": "2m"
                            },
                            "callbackRequestTimeout": {
                                    "connectionAndInactivity": "10m",
                                    "wholeCycle": "10m"
                            },
                            "healthcheckfilepath": "../public/healthcheck.docx",
                            "savetimeoutdelay": 5000,
                            "edit_singleton": false,
                            "forgottenfiles": "forgotten",
                            "forgottenfilesname": "output",
                            "maxRequestChanges": 20000,
                            "openProtectedFile": true,
                            "isAnonymousSupport": true,
                            "editorDataStorage": "editorDataMemory",
                            "assemblyFormatAsOrigin": true,
                            "newFileTemplate" : "../../document-templates/new",
                            "downloadFileAllowExt": ["pdf", "xlsx"],
                            "tokenRequiredParams": true
                    },

“…/public/healthcheck.docx” - what is it? Is this something that should work but doesn’t?

Hello @Sargatanas
Sorry for the late reply. The provided logs contain such lines:
[AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672
This port is used by RabbitMQ service. Please make sure that this port isn’t occupied by other app or your proxy server doesn’t interrupt this scenario. Additionally, please check out that the service itself is running: service rabbitmq-server status
About the ports: Ports which must be opened for ONLYOFFICE Docs - ONLYOFFICE

RabbitMQ is not installed and has never been installed. The server is used exclusively for online editing of documents from nextcloud. This is how the previous employee installed it and everything worked.
It seems to me that you are going down the wrong path.

It shouldn’t work without this service: ONLYOFFICE Docs for Linux system requirements - ONLYOFFICE

Let’s start from scratch. Please point me to the guide which colleague of yours used for installation. Additionally specify your OS and when exactly the issue occurred. Was it in the first day of installation or after some actions on the server side?
One more thing. A few days ago we released Document server v.7.5. Please update your installation to the latest version before continuing troubleshooting.

The Rabbit service is required to move messages between the server and the client. If you can see the installation guide, the service is necessary so check the port. In addition to what you are referring to “/healthcheck.docx” it is a complement so to speak, to enter the address through the web and see if the system is working similar to the page where it says “welcome”

2 Likes

Having identical symptoms after updating OnlyOffice to 7.5 (enterprise edition). Example page works fine after restart, but any hits to /healthcheck results in a 502 bad gateway.

The IP address where the service is installed, if it has access to the Internet? or does it have any blocking by a firewall?

Previously I left the IP where the Documentserver is installed without internet access. And the same thing happened to me, now I always leave it with an Internet connection but protected with a firewall.

…Uninstalling and installing again helped…

I’m having the same issue with a similar setup. I have 4 different machines running; Nextcloud, OnlyOffice, OpenMediaVault, nginx. They are all on individual Ubuntu server 22.04 virtuals on an ESXi box. Working fine for approx a year. Issue came up with a routine update using apt update. I’m reverted to a snapshot and tested and have found that updating the OnlyOffice machine is the one that breaks the connection. I know enough linux to get me in trouble and nothing on how to get me out… I’m thinking of upgrading one package at a time to point to the offending. Sound time consuming… any other ideas?

EDIT:
OK, I ran through the majority of updates with no issues. The last group of updates where onlyoffice-documentserver rabbitmq-server and a few held back packages. I intended to install just the held back packages and then upgrade OO and rabbit but I inadvertently ran them together. During the OO upgrade a user “interactive pause” was initiated for the configuration file /etc/onlyoffice/documentserver-example/default.json file is different than the one from the distributor. In previous updates that failed I had opted to keep the existing version, this round I installed the package maintainer’s version. Updates completed, I rebooted system and we’re back to a working system. In closing, I’m thinking that file is the one to look at if you’re still having issues.

My terminal history starting with the “interactive pause”.

Configuration file ‘/etc/onlyoffice/documentserver/default.json’
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer’s version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** default.json (Y/I/N/O/D/Z) [default=N] ? d

BEGIN COMPARISON
— /etc/onlyoffice/documentserver/default.json 2023-03-01 19:50:08.177854453 +0000
+++ /etc/onlyoffice/documentserver/default.json.dpkg-new 2023-11-05 00:06:18.000000000 +0000
@@ -49,7 +49,6 @@
“connectOptions”: {
“port”: 5672,
“host”: “localhost”,

  •                   "name": "admin",
                      "reconnect": false
              },
              "queueconverttask": "ds.converttask",
    

@@ -87,11 +86,11 @@
“favIconUrlSlide” : “/web-apps/apps/presentationeditor/main/resources/img/favicon.ico”,
“fileInfoBlockList” : [“FileUrl”],
“pdfView”: [“pdf”, “djvu”, “xps”, “oxps”],

  •           "wordView": ["doc", "dotx", "dotm", "dot", "fodt", "ott", "rtf", "mht", "html", "htm", "xml", "epub", "fb2"],
    
  •           "wordView": ["doc", "dotx", "dotm", "dot", "fodt", "ott", "rtf", "mht", "mhtml", "html", "htm", "xml", "epub", "fb2", "sxw", "stw", ">
              "wordEdit": ["docx", "docm", "docxf", "oform", "odt", "txt"],
    
  •           "cellView": ["xls", "xlsb", "xltx", "xltm", "xlt", "fods", "ots"],
    
  •           "cellView": ["xls", "xlsb", "xltx", "xltm", "xlt", "fods", "ots", "sxc", "xml", "et", "ett"],
              "cellEdit": ["xlsx", "xlsm", "ods", "csv"],
    
  •           "slideView": ["ppt", "ppsx", "ppsm", "pps", "potx", "potm", "pot", "fodp", "otp"],
    
  •           "slideView": ["ppt", "ppsx", "ppsm", "pps", "potx", "potm", "pot", "fodp", "otp", "sxi", "dps", "dpt"],
              "slideEdit": ["pptx", "pptm", "odp"],
              "publicKey": "long ass key #1>
              "modulus": "long ass key #2>
    

@@ -106,6 +105,7 @@
“tenants”: {
“baseDir”: “”,
“baseDomain”: “”,

  •           "filenameConfig": "config.json",
              "filenameSecret": "secret.key",
              "filenameLicense": "license.lic",
              "defaultTenant": "localhost",
    

— /etc/onlyoffice/documentserver/default.json 2023-03-01 19:50:08.177854453 +0000
+++ /etc/onlyoffice/documentserver/default.json.dpkg-new 2023-11-05 00:06:18.000000000 +0000
@@ -49,7 +49,6 @@
“connectOptions”: {
“port”: 5672,
“host”: “localhost”,

  •                   "name": "admin",
                      "reconnect": false
              },
              "queueconverttask": "ds.converttask",
    

@@ -87,11 +86,11 @@
“favIconUrlSlide” : “/web-apps/apps/presentationeditor/main/resources/img/favicon.ico”,
“fileInfoBlockList” : [“FileUrl”],
“pdfView”: [“pdf”, “djvu”, “xps”, “oxps”],

  •           "wordView": ["doc", "dotx", "dotm", "dot", "fodt", "ott", "rtf", "mht", "html", "htm", "xml", "epub", "fb2"],
    
  •           "wordView": ["doc", "dotx", "dotm", "dot", "fodt", "ott", "rtf", "mht", "mhtml", "html", "htm", "xml", "epub", "fb2", "sxw", "stw", ">
              "wordEdit": ["docx", "docm", "docxf", "oform", "odt", "txt"],
    
  •           "cellView": ["xls", "xlsb", "xltx", "xltm", "xlt", "fods", "ots"],
    
  •           "cellView": ["xls", "xlsb", "xltx", "xltm", "xlt", "fods", "ots", "sxc", "xml", "et", "ett"],
              "cellEdit": ["xlsx", "xlsm", "ods", "csv"],
    
  •           "slideView": ["ppt", "ppsx", "ppsm", "pps", "potx", "potm", "pot", "fodp", "otp"],
    
  •           "slideView": ["ppt", "ppsx", "ppsm", "pps", "potx", "potm", "pot", "fodp", "otp", "sxi", "dps", "dpt"],
              "slideEdit": ["pptx", "pptm", "odp"],
              "publicKey": "long ass key #1>
              "modulus": "long ass key #2>
    

@@ -106,6 +105,7 @@
“tenants”: {
“baseDir”: “”,
“baseDomain”: “”,

  •           "filenameConfig": "config.json",
              "filenameSecret": "secret.key",
              "filenameLicense": "license.lic",
              "defaultTenant": "localhost",
    

@@ -138,11 +138,13 @@
“forgottenfilesname”: “output”,
“maxRequestChanges”: 20000,
“openProtectedFile”: true,

  •                           "isAnonymousSupport": true,
                              "editorDataStorage": "editorDataMemory",
                              "assemblyFormatAsOrigin": true,
                              "newFileTemplate" : "../../document-templates/new",
                              "downloadFileAllowExt": ["pdf", "xlsx"],
    
  •                           "tokenRequiredParams": true
    
  •                           "tokenRequiredParams": true,
    
  •                           "allowPrivateIPAddressForSignedRequests": true
                      },
                      "requestDefaults": {
                              "headers": {
    

@@ -150,7 +152,7 @@
“Connection”: “Keep-Alive”
},
“gzip”: true,

  •                           "rejectUnauthorized": false
    
  •                           "rejectUnauthorized": true
                      },
                      "autoAssembly": {
                              "enable": false,
    

@@ -160,7 +162,7 @@
“utils”: {
“utils_common_fontdir”: “null”,
“utils_fonts_search_patterns”: “.ttf;.ttc;*.otf”,

  •                           "limits_image_types_upload": "jpg;jpeg;jpe;png;gif;bmp"
    
  •                           "limits_image_types_upload": "jpg;jpeg;jpe;png;gif;bmp;svg"
                      },
                      "sql": {
                              "type": "postgres",
    

@@ -174,17 +176,23 @@
“charset”: “utf8”,
“connectionlimit”: 10,
“max_allowed_packet”: 1048575,

  •                           "pgPoolExtraOptions": {}
    
  •                           "pgPoolExtraOptions": {},
    
  •                           "damengExtraOptions": {}
                      },
                      "redis": {
                              "name": "redis",
                              "prefix": "ds:",
    
  •                           "host": "localhost",
    
  •                           "host": "127.0.0.1",
                              "port": 6379,
    
  •                           "options": {}
    
  •                           "options": {},
    
  •                           "optionsCluster": {},
    
  •                           "iooptions": {},
    
  •                           "iooptionsClusterNodes": [
    
  •                           ],
    
  •                           "iooptionsClusterOptions": {}
                      },
                      "pubsub": {
    
  •                           "maxChanges": 0
    
  •                           "maxChanges": 1000
                      },
                      "expire": {
                              "saveLock": 60,
    

@@ -195,6 +203,7 @@
“message”: 86400,
“lastsave”: 604800,
“forcesave”: 604800,

  •                           "forcesaveLock": 5000,
                              "saved": 3600,
                              "documentsCron": "0 */2 * * * *",
                              "files": 86400,
    

@@ -214,10 +223,11 @@
“errorcode”: 403
},
“request-filtering-agent” : {

  •                           "allowPrivateIPAddress": true,
    
  •                           "allowMetaIPAddress": true
    
  •                           "allowPrivateIPAddress": false,
    
  •                           "allowMetaIPAddress": false
                      },
                      "secret": {
    
  •                           "browser": {"string": "secret", "file": ""},
                              "inbox": {"string": "secret", "file": ""},
                              "outbox": {"string": "secret", "file": ""},
                              "session": {"string": "secret", "file": ""}
    

@@ -230,6 +240,9 @@
“outbox”: false
}
},

  •                           "browser": {
    
  •                                   "secretFromInbox": true
    
  •                           },
                              "inbox": {
                                      "header": "Authorization",
                                      "prefix": "Bearer ",
    

END OF COMPARISON
Configuration file ‘/etc/onlyoffice/documentserver/default.json’
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer’s version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** default.json (Y/I/N/O/D/Z) [default=N] ? y
Installing new version of config file /etc/onlyoffice/documentserver/default.json …
Installing new version of config file /etc/onlyoffice/documentserver/development-linux.json …
Installing new version of config file /etc/onlyoffice/documentserver/development-mac.json …
Installing new version of config file /etc/onlyoffice/documentserver/development-windows.json …
Installing new version of config file /etc/onlyoffice/documentserver/logrotate/ds.conf …
Installing new version of config file /etc/onlyoffice/documentserver/nginx/includes/ds-docservice.conf …
Installing new version of config file /etc/onlyoffice/documentserver/nginx/includes/http-common.conf …
Generating AllFonts.js, please wait…Done
Generating presentation themes, please wait…Done
Generating js caches, please wait…Done
Installing plugins, please wait…Done
Congratulations, the ONLYOFFICE documentserver has been installed successfully!

1 Like