Cert Manager with Let's Encrypt¶
Imported from Confluence
Content may be outdated. Verify before following any procedures. View original | Last updated: May 2024
Used articles and documentation¶
- How to use cert-manager to create certificates from Let's Encrypt: Ssl Certificates With Cert Manager In Kubernetes
- How to delegate domain: Dns01
Task Description¶
In GCP shared account ss-shared-fyber-dev we deployed private dns zone ofw-dev-dns01.gke.fyber.com which is used to expose gke services through nginx controller with GCP Private TCP Load Balancer. Because GCP TCP Internal Load Balancer doesn't work with Google Managed certificates we should terminate ssl on nginx controller. In this task we should create a ssl certificate, put it in kubernetes secret and use this secret in nginx controller.
How-To¶
- Certificate should be created and renewed automatically - to automate the process of creating and renewing of certificates we decided to use combination of cert-manager and Let's Encrypt deployed in every GKE cluster
- Validation for private subnets and private dns zones
- Private subnet - we can't use HTTP-01 since nginx is not available from internet, therefore we decided to use DNS-01
-
Private DNS zone - in order to validate certificate with DNS-01 we should create a TXT record under domain name we used in certificate and Let's Encrypt should be able resolve this record. Since we are using private zone the record is not available from outside GCP private subnet. To solve this problem we delegated private domain to public domain and gave cert-manager permissions to create records in this public domain:
- Delegating private domain gke-core-offerwall-dev-useast1.fyber.com`:
- Create public dns zone gke-offerwall-dev-dns01.fyber.com. in shared vpc and delegate it from dnsmadeeasy. This zone will be used for delegating our private domain validation TXT records
- In dnsmadeeasy create CNAME record _acme-challenge.gke-core-offerwall-dev-useast1.fyber.com. point to _acme-challenge.gke-offerwall-dev-dns01.fyber.com.
- Make sure cert-manager in GKE can resolve this record. In our case it couldn't with default ns servers. This problem was solved by configuring cert-manager to use ns 8.8.8.8`
Example of helm deployments¶
cert-manager - appgrowthplatform (Gitlab)
issuer and certificate - appgrowthplatform (Gitlab)
When you do new deployment, set .Values.issuer.api to staging, otherwise Let's Encrypt can block you if you have lot of unsuccessful attempts:
- test that order is valid with stating api
- destroy release with staging api, which will clean all orders and certs (helmfile ... destroy)
- deploy with production api
In GCP shared account ss-shared-fyber-dev we deployed private dns zone ofw-dev-dns01.gke.fyber.com which is used to expose gke services through nginx controller with GCP Private TCP Load Balancer. Because GCP TCP Internal Load Balancer doesn't work with Google Managed certificates we should terminate ssl on nginx controller. In this task we should create a ssl certificate, put it in kubernetes secret and use this secret in nginx controller.
How-To¶
- Certificate should be created and renewed automatically - to automate the process of creating and renewing of certificates we decided to use combination of cert-manager and Let's Encrypt deployed in every GKE cluster
- Validation for private subnets and private dns zones
- Private subnet - we can't use HTTP-01 since nginx is not available from internet, therefore we decided to use DNS-01
-
Private DNS zone - in order to validate certificate with DNS-01 we should create a TXT record under domain name we used in certificate and Let's Encrypt should be able resolve this record. Since we are using private zone the record is not available from outside GCP private subnet. To solve this problem we delegated private domain to public domain and gave cert-manager permissions to create records in this public domain:
- Delegating private domain gke-core-offerwall-dev-useast1.fyber.com`:
- Create public dns zone gke-offerwall-dev-dns01.fyber.com. in shared vpc and delegate it from dnsmadeeasy. This zone will be used for delegating our private domain validation TXT records
- In dnsmadeeasy create CNAME record _acme-challenge.gke-core-offerwall-dev-useast1.fyber.com. point to _acme-challenge.gke-offerwall-dev-dns01.fyber.com.
- Make sure cert-manager in GKE can resolve this record. In our case it couldn't with default ns servers. This problem was solved by configuring cert-manager to use ns 8.8.8.8`
Example of manual provisioning¶
You need to ssh to one of our instances which already has installed needed tool - devops.vm-offerwall-prod-useast1.fyber.com
root@vm-devops-offerwall-prod-useast1-01-0:~/cert# certbot certonly --manual --preferred-challenges dns -d "*.mmp.digitalturbine.com"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for mmp.digitalturbine.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.
Are you OK with your IP being logged?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please deploy a DNS TXT record under the name
_acme-challenge.mmp.digitalturbine.com with the following value:
XaSA5yc8LV7NuzWmpQqO2ga
Before continuing, verify the record is deployed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
Waiting for verification...
Cleaning up challenges
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/mmp.digitalturbine.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/mmp.digitalturbine.com/privkey.pem
Your cert will expire on 2024-08-05. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew *all* of your certificates, run
"certbot renew"
- If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le