Thank you for the logs.
There’re a lot of 5672 port error entries (RabbitMQ). Also I see websocket issue from browser log. Do you have proxy server in front of Document server? It looks like something interrupts the situation.
We were able to resolve the the issue with RabbitMQ on our production server and now we are left with one error from the docservice out.log that we feel is the main error and cause of our issue.
[2023-02-13T09:17:57.148] [ERROR] nodeJS - sendServerRequest error: docId = 72bce0c5-aa75-f143-4c2e-9a5727c26732_48563;url = https:// websiteurl:8443/o/onlyoffice/doc?key=UlRhOGhkV0dMWEFYMHMzZ3NBdjduaUNYQjRaRDJ1Q0VPSWxGWGxNSk1kaz0_MzEzMzg1NQ==;data = {“key”:“72bce0c5-aa75-f143-4c2e-9a5727c26732_48563”,“status”:2,“url”:“https:// websiteurl:2443/cache/files/72bce0c5-aa75-f143-4c2e-9a5727c26732_48563_6153/output.xlsx/output.xlsx?md5=YP6uwp6IA3dmwWmgnEGqMw&expires=1676298778&disposition=attachment&filename=output.xlsx”,“changesurl”:“https:// websiteurl:2443/cache/files/72bce0c5-aa75-f143-4c2e-9a5727c26732_48563_6153/changes.zip/changes.zip?md5=-f4X8ZlXwchaiz-86sAUhQ&expires=1676298778&disposition=attachment&filename=changes.zip”,“history”:{“serverVersion”:“7.3.0”,“changes”:[{“created”:“2022-10-31 19:05:32”,“user”:{“id”:“40022”,“name”:“User Name”}}]},“users”:[“40022”],“actions”:[{“type”:0,“userid”:“40022”}],“lastsave”:“2023-02-10T11:53:07.464Z”,“notmodified”:false}
Error: Error response: statusCode:500; headers:{“x-content-type-options”:“nosniff”,“x-frame-options”:“SAMEORIGIN”,“x-xss-protection”:“1”,“set-cookie”:[“JSESSIONID=64E83F18042FF65EA7EB07F14D362617; Path=/; Secure; HttpOnly”],“content-length”:“211”,“date”:“Mon, 13 Feb 2023 14:17:57 GMT”,“connection”:“close”}; body:
{
“error”: 1,
“message”: “Couldn’t download or save file: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”
}
at Request._callback (/snapshot/server/build/server/Common/sources/utils.js:0:0)
at Request.init.self.callback (/snapshot/server/build/server/Common/node_modules/request/request.js:185:22)
at Request.emit (events.js:198:13)
at Request. (/snapshot/server/build/server/Common/node_modules/request/request.js:1154:10)
at Request.emit (events.js:198:13)
at IncomingMessage. (/snapshot/server/build/server/Common/node_modules/request/request.js:1076:12)
at Object.onceWrapper (events.js:286:20)
at IncomingMessage.emit (events.js:203:15)
at endReadableNT (_stream_readable.js:1143:12)
at process._tickCallback (internal/process/next_tick.js:63:19)
[2023-02-13T09:17:57.148] [WARN] nodeJS - commandSfcCallback backoff limit exceeded: docId = 72bce0c5-aa75-f143-4c2e-9a5727c26732_48563
[2023-02-13T09:17:57.149] [WARN] nodeJS - storeForgotten: docId = 72bce0c5-aa75-f143-4c2e-9a5727c26732_48563
We see that the file is moved to Forgotten and stored there after the sendServerRequest error. It appears to deal with certificates however it looks more like it doesn’t know where to look for them rather than a problem with the certificates themselves. The only thing I’ve changed is the website url and the username from this set of log entries.
“error”: 1,
“message”: “Couldn’t download or save file: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”
Could you please run a test? Go to Document server > /etc/onlyoffice/documentserver/default.json > find this parameter: "rejectUnauthorized": true and change it to false. After that restart all Document server services with supervisor or systemd command and re-check the situation.
NB! We strongly recommend to prepare whole server backup before any server side actions.
It was set to false previously. Underneath the requestDefaults there is “rejectUnauthorized”: false. To my knowledge this parameter has not been changed any time recently and has gone through multiple server restarts as well as service restarts.
It seems that the issue is still related to your certificates. We found similar description here: How to resolve PKIX path building failed/unable to find valid certification path to requested target issue? - Apisero
Please double check Document server certificate. If it has open access to the Internet, you can use any online SSL-checkers, for example: SSL Checker
The issue in general is related to the fact that Liferay doesn’t accept Document server certificate.
Looking further into the issue we tried using InstallCerts.java to install the certificate we were using however that did not work for our environment, I then ran the command:
./keytool -importcert -alias someAliasName -keystore {JAVA_HOME}/lib/security/cacerts -file {PathToDownloads}/certificateFileName.crt
which asked for the keystore password and then asked if I would like to trust the certificate. After restarting the Liferay server I was able to update documents where the changes were saved automatically and versions were updated. This appears to have resolved every issue we were having. To my understanding, it looks like that while we updated the SSL certificates for both Liferay and OnlyOffice document server, the installed version of Java did not trust the installed certificates and it had to be added to Java’s trusted certificate store.
Unrelated to the fix but the Qualys SSL checker did say that our site was missing its intermediate certificate which we promptly fixed as well.
We’re glad that the situation is solved. Please feel free to contact us if you face any issues.