There I have similar problem.
Certs are created by XCA app. These certs works from browser (edge and chrome) on PC, browser on Android (chrome), only office app on PC. The same CA i use for VPN (both server and clients), Tapatalk server, Unifi.
When I try to connect by only office app on Android I got SSL handshake error both on app and in the log on haproxy on the front.
I do many tests with another test certs, another root CA (issued by XCA). Always the same. SSL handshake error.
Today I found only office projects app on Google Play. So I try it, and. With this app I successfully connect to server. There an app could open all my documents without problems. When I back to only office documents app I have connection to my server (magic!), I can connect open docs.
So, in my opinion, there is the bug - first validation of certificates should be corrected in only office app for Android.