ArgoCD behind Istio on Rancher

This article has been updated to reflect Argo v2.4.4. Enjoy!

Implementing Argo in my lab cluster and ran into some head aches. There’s a bunch of ingress configurations documented on the installation guide but Istio is not one of them so I figured I’d document it here.

Basing this article on these:
https://github.com/argoproj/argo-cd/issues/2784
https://gist.github.com/janeczku/b16154194f7f03f772645303af8e9f80

In order to make it work you’ll going to have to rename some of the service ports and start the API server with the –insecure flag. It sounds scary, but since you have the Istio envoy sidecar in your pod the traffic will be encrypted either way so it does not matter.

See something that is wrong or that can be improved. Please leave a comment and I’ll update the instructions!

Installing Argo

Prepare the name space

Create the namespace and label it in order to enable automatic injection (injection optional but recommended).

kubectl create namespace argocd
kubectl label namespace argocd istio-injection=enabled --overwrite

Then follow the installation instructions here:

https://argoproj.github.io/argo-workflows/quick-start/

Get the modifications

git clone https://github.com/epacke/argo-istio .

What you have downloaded

ConfigMap.yaml

Tells argo-server to start in “insecure mode”.

Gateway.yaml

This would be the Istio Gateway. If you have a gateway already you can skip this definition. And in that case I assume you know what to do.

Services.yaml

Same as the original services, but the port names has been prefixed with “http” to get proper stats in Kiali.

VirtualService.yaml

A way to configure the Istio gateway to send traffic using a specific host header to a specific service.

kustomization.yaml

Ties all the YAML files together. My previous article used this file to patch the existing services but Kustomize has changed the syntax since then and I am currently watching Star Wars Bobba Fett while writing this article to converting it to the new syntax just took too much concentration… 😉

Configure your domain

Edit both VirtualService.yaml and Gateway.yaml to use your domain instead of argocd.xip.io. Only two places needs to be modified.

Apply the modifications

kubectl apply -k .

Test

Now you can surf to your Istio ingress port and test it out. The user is admin and the password is the same as your argocd-server pod. The commands below will show you the password and get the Istio ingress gateway node port. Just remember to replace argocd.xip.io with your domain.

echo "Argo admin password is \"$(kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2)\""
export SECURE_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}')
argocd --grpc-web login argocd.xip.io:$SECURE_INGRESS_PORT

You should also be able to surf to https://<your domain>:$SECURE_INGRESS_PORT using your browser.

Adding Rancher cluster credentials

2022-07-08 Update – Ranchers interface has changed a bit but it’s close enough so I won’t change the screenshots.

This is section is applicable if you, like me, are running Rancher. In that case adding the cluster credentials won’t work like it usually does. Instead, you need to do this manually.

Creating the user and assigning permissions

First we need to create a user. Go to Global -> Security -> Users.

Click on Add and then proceed to create a Standard user with a name of your choosing. For the purpose of this guide we’ll use argo-service. Assign a password and Click on Create.

Next, navigate to the User Cluster you want Argo to be able to access.

Click on Members.

Then click on Add Member. Search for argo-service, assign the role “Member” and click on Create. Worth noting, I’ve had mixed success here and I am still not sure why. Second time around I had to give the user Cluster Owner, but I’m sure this is customize-able, I just won’t spend time doing it in my lab.

Creating an API key

1. Login as argo-service and choose API & Keys from the top right corner
2. Click on Add Key
3. Add a description if you want to and leave everything as default
4. Click on Create
5. Copy all the information at the page into a password storage of some kind and click on Close

Creating the Argo Cluster secret

Before you create these secrets you need to determine the URL to your user cluster. You can get this by clicking on Cluster on the top menu in Rancher and then click on Kubeconfig File.

The URL is marked with red below:

Next, create the following YAML file. Note that the server property is the URL from above and the bearerToken is the bearer token you got when creating the API key before. This configuration assumes that you have a legitimate certificate for your rancher cluster. If you don’t I have a guide for setting this up here.

apiVersion: v1
kind: Secret
metadata:
  name: rancherprod-cluster-secret
  labels:
    argocd.argoproj.io/secret-type: cluster
  namespace: argocd
type: Opaque
stringData:
  name: rancher-prod
  server: https://rancher.xip.io/k8s/clusters/c-aabb12
  config: |
    {
      "bearerToken": "token-123aa:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
      "tlsClientConfig": {
        "insecure": false
      }
    }

Run kubectl apply -f <file name> to create the secret. Argo should automatically pick this up and have access to your user cluster.

Troubleshooting

When does things go as they should in IT. More or less never. Argo has this great guide on how to troubleshoot adding cluster credentials. You can find it here.

The only thing I’d like to add that was not super clear to me is the following things:

1. Make sure to look at the right version of the docs. The command to generate the kubeconfig was argocd-util kubeconfig in my version, not argocd-util cluster kubeconfig as it is in other versions.
2. The api-server-url is the Rancher user cluster URL, ie https://rancher.xip.io/k8s/clusters/c-aabb12