Virtual-Path configuration für Kubernetes installation

uwohlfeil · 7 November 2021 21:44

Hi,

I just test a Kubernetes Installation based on you documentation at https://github.com/ONLYOFFICE/Kubernetes-Docs and it works fine.

I used the nginx-ingress as described here: Expose DocumentServer via HTTP

But is there a way to expose it to a subdirectory like you described it here for a normal installation?

{{ if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: documentserver
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 100m
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Forwarded-Host     $best_http_host/onlyoffice/;
spec:
  {{ if .Values.ingress.ssl.enabled }}
  tls: 
  - hosts:
    - {{ .Values.ingress.ssl.host }}
    secretName: {{ .Values.ingress.ssl.secret }}
  {{ end }}
  rules:
  {{ if .Values.ingress.ssl.enabled }}
  - host: {{ .Values.ingress.ssl.host }}
  {{ else }}
  - host:
  {{ end }}
    http:
      paths:
      - path: /onlyoffice(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: documentserver
            port:
              number: 8888
{{ end }}

But it always redirects me to /welcome.

The Variable $best_http_host seems to be the dynamic hostname from the lua script in the generated nginx config in the nginx-ingress.

Best regards
Uli

Carl · 8 November 2021 14:01

Hi Uli,

Sorry, but we have no example configuration for virtual path in Nginx Ingress Controller.
As far as I understand, there were some issues with X-Forwarded-* headers which prevented us from creating such example config.

uwohlfeil · 8 February 2022 17:16

Hi Carl,

is it possible to modify your internal nginx configuration to support optimal the x-forwarded-prefix header?

Your relevant configuration in the kubernetes installation seems to look like this:

/tmp/proxy_nginx/includes/http-common.conf

map $http_host $this_host {
    "" $host;
    default $http_host;
}

map $http_x_forwarded_proto $the_scheme {
     default $http_x_forwarded_proto;
     "" $scheme;
}

map $http_x_forwarded_host $the_host {
    default $http_x_forwarded_host;
    "" $this_host;
}

map $http_upgrade $proxy_connection {
  default upgrade;
  "" close;
}

proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Forwarded-Host $the_host;
proxy_set_header X-Forwarded-Proto $the_scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

The trick seems to be, that for virtual hosts, the path is a part of the x-Forwarded-Host.
For example:
Incoming Header: x-Forwarded-Host: demosystem.ch/onlyoffice
Transfered Header: x-Forwarded-Host: demosystem.ch/onlyoffice

The nginx will transfer this header unchanged to the docservice etc.
But maybe it is possible to merge the X-Forwarded-Host with the X-Forwarded-Prefix to the docservice so the Revers Proxy in the front can transfer the path information also as separate header.

For example:
Incoming Header: x-Forwarded-Host: demosystem.ch
Incoming Header: x-Forwarded-Prefix: /onlyoffice
Transfered Header: x-Forwarded-Host: demosystem.ch/onlyoffice

So i think the config has to be changed to something like this:

map $http_x_forwarded_host $the_host {
    default $http_x_forwarded_host$http_x_forwarded_prefix;
    "" $this_host;
}

Background is, that Kubernetes ingress seems to support the X-Forwarded-Prefix better then the modified X-Forwarded-Host (as far as I understand, a kubernetes internal nginx lua script recalculate the X-Forwarded-Host header).

I test this in a “normal” windows installation and changed the Reverse Proxy to this header and it seems to work correctly in both cases (modified host and additional prefix).

Do you already check this solution?
Do this change has any disadvantages?

uwohlfeil · 10 February 2022 11:56

Hello together,

I just verified my guess. I create a own Dockerfile based on your onlyoffice/docs-proxy-de, just replace the file /etc/nginx/includes/http-common.conf and pushed the container on a private repository for testing.

Docker

DockerFile

FROM onlyoffice/docs-proxy-de:6.4.1.45

COPY http-common.conf /etc/nginx/includes/

http-common.conf

include /tmp/http-upstream.conf;
map $http_host $this_host {
    "" $host;
    default $http_host;
}

map $http_x_forwarded_proto $the_scheme {
     default $http_x_forwarded_proto;
     "" $scheme;
}

map $http_x_forwarded_host $the_host {
    default $http_x_forwarded_host$http_x_forwarded_prefix;
    "" $this_host;
}

map $http_upgrade $proxy_connection {
  default upgrade;
  "" close;
}

proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Forwarded-Host $the_host;
proxy_set_header X-Forwarded-Proto $the_scheme;

The only change in the file is the following line:
default $http_x_forwarded_host$http_x_forwarded_prefix;

Helm

Then on Helm side, I changed the ingresses\documentserver.yaml to the following:

ingresses\documentserver.yaml

{{ if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: documentserver
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 100m
    nginx.ingress.kubernetes.io/x-forwarded-prefix: /onlyoffice
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  {{ if .Values.ingress.ssl.enabled }}
  tls: 
  - hosts:
    - {{ .Values.ingress.ssl.host }}
    secretName: {{ .Values.ingress.ssl.secret }}
  {{ end }}
  rules:
  {{ if .Values.ingress.ssl.enabled }}
  - host: {{ .Values.ingress.ssl.host }}
  {{ else }}
  - host:
  {{ end }}
    http:
      paths:
      - path: /onlyoffice(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: documentserver
            port:
              number: 8888
{{ end }}

I changed the following lines:

...
    nginx.ingress.kubernetes.io/x-forwarded-prefix: /onlyoffice
    nginx.ingress.kubernetes.io/rewrite-target: /$2
...
      - path: /onlyoffice(/|$)(.*)

It will be very nice if you can check and at my fix into your version of the container if possible.

Best regards
Uli

Carl · 11 February 2022 08:57

Hello Uli,

Thank you, we will take a look at this config file.

cwrau · 11 October 2022 10:48

Hey, did you get this working?

uwohlfeil · 12 October 2022 08:20

@cwrau I still use a modified version of the original docker container and update it for each version. My docker image based on the original and just replace the configuration file.
I can post you a example for this if you need it.

@ONLYOFFICE: Is this still a open issue?

cwrau · 12 October 2022 12:01

Yeah, the example would be great

uwohlfeil · 12 October 2022 13:02

Dockerfile

docs-proxy-de (docs-proxy-de Developer / docs-proxy-ee Enterprise) is the version you want to build.
7.1.1-4 Is the version of ONLYOFFICE you want to base on.

FROM onlyoffice/docs-proxy-de:7.1.1-4
COPY http-common.conf /etc/nginx/includes/

http-common.conf (Modified as described)

include /tmp/http-upstream.conf;
map $http_host $this_host {
    "" $host;
    default $http_host;
}

map $http_x_forwarded_proto $the_scheme {
     default $http_x_forwarded_proto;
     "" $scheme;
}

map $http_x_forwarded_host $the_host {
    default $http_x_forwarded_host$http_x_forwarded_prefix;
    "" $this_host;
}

map $http_upgrade $proxy_connection {
  default upgrade;
  "" close;
}

proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Forwarded-Host $the_host;
proxy_set_header X-Forwarded-Proto $the_scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

I build this via the local docker environment and push it to our Gitlab.
I think you can do the same thing with dockerhub and so on.

docker build -t registry.our-server:443/username/docs-proxy-de:7.1.1-4 .  
docker push registry.our-server:443/username/docs-proxy-de:7.1.1-4

The last step is to replace the original docker container with your version in the values.yaml file of the ONLYOFFICE helm configuration.

...
...
proxy:
  proxyContainerImage: registry.our-server/username/docs-proxy-de:7.1.1-4
...
...

In our case we also need a authentication for our non open container registry.

But this is just a workaround and I hope ONLYOFFICE will fix this in a near future

Alexandre · 13 October 2022 14:50

Hello @uwohlfeil and @cwrau
We are still working on it. As far as I know, there’s open ticket in Zendesk which was created by @uwohlfeil. My colleagues are still working on it and we will notify you when we have something to share.
Sorry for inconvenience.

uwohlfeil · 1 February 2023 09:19

Hallo @Alexandre ,

do you have any news about this issue?

Regards Uli

Alexandre · 2 February 2023 10:09

Hello @uwohlfeil
Please accept our sincere apologies. We are still working on it. My colleagues will contact you via Zendesk system when we have something to share.

davide.cavestro · 13 February 2023 09:47

Just hit the same issue… I suppose it affects every deployment of the document server container image behind a reverse proxy (i.e. nginx).
I am wondering if the config examples and related docs are valid for this kind of deployment.

Alexandre · 14 February 2023 12:26

Hello @davide.cavestro
We’re still looking into it. We will provide a solution as soon as possible.

Alexandre · 15 February 2023 10:42

Probably we found workaround solution. This is a temporary solution while we’re continuing to troubleshoot the situation, but it works.
We used mentioned http-common.conf file as an example and we changed it a little bit:

include /tmp/http-upstream.conf;
map $http_host $this_host {
    "" $host;
    default $http_host;
}

map $http_x_forwarded_proto $the_scheme {
     default $http_x_forwarded_proto;
     "" $scheme;
}

map $http_x_forwarded_host $the_host {
    default $http_x_forwarded_host$http_x_forwarded_prefix;
    "" $this_host;
}

map $http_upgrade $proxy_connection {
  default upgrade;
  "" close;
}

proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Forwarded-Host $the_host;
proxy_set_header X-Forwarded-Proto $the_scheme;
proxy_set_header X-Forwarded-Prefix "/";

Also you have to add a few lines to values.yaml config file to annotations section.
Keep default lines and just add this:

nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/x-forwarded-prefix: /ds
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Forwarded-Host     $best_http_host/ds;

Also go to templates/ingress/documentserver.yaml and add this line to -path: parameter:
/ds(/|$)(.*)

Please pay attention to ‘/ds’ value in my examples. You have to change it to your virtual path.
One more thing, we can’t guarantee that integrated example will work properly when these actions are performed. Also please prepare whole server backup\snapshot before any actions just in case.

davide.cavestro · 16 February 2023 16:43

Thanks, will try with the real app, as it fails for the example.
Please also consider passing the incoming headers when the ingress is trusted, as per proxy_pass_request_headers on.

davide.cavestro · 16 February 2023 17:16

I confirm it works like a charm on the iframe within the app.

Alexandre · 17 February 2023 14:34

We’re glad to be of service.