Part 02: Building a Sovereign Software Factory: The Local Root CA & Trust Chains

Source: Dev.to | Dec 16, 2025 (UTC)

GitHub: https://github.com/InfiniteConsult/0006_cicd_part02_certificate_authority

TL;DR: In this installment, we solve the “Trust Gap” inherent in self-hosted infrastructure. We reject the insecurity of HTTP and the limitations of public CAs (Let’s Encrypt) by building our own Local Certificate Authority. We use openssl to mint a Root CA, automate the issuance of SAN-compliant certificates for every service, and manually inject trust into the fragmented keystores of our Host OS, Browsers, and Java applications to eliminate SSL handshake failures.

The Sovereign Software Factory Series:

  1. Part 01: Building a Sovereign Software Factory: Docker Networking & Persistence
  2. Part 02: Building a Sovereign Software Factory: The Local Root CA & Trust Chains (You are here)
  3. Part 03: Building a Sovereign Software Factory: Self-Hosted GitLab & Secrets Management
  4. Part 04: Building a Sovereign Software Factory: Jenkins Configuration as Code (JCasC)
  5. Part 05: Building a Sovereign Software Factory: Artifactory & The “Strict TLS” Trap
  6. Part 06: Building a Sovereign Software Factory: SonarQube Quality Gates
  7. Part 07: Building a Sovereign Software Factory: ChatOps with Mattermost
  8. Part 08: Building a Sovereign Software Factory: Observability with the ELK Stack
  9. Part 09: Building a Sovereign Software Factory: Monitoring with Prometheus & Grafana
  10. Part 10: Building a Sovereign Software Factory: The Python API Package (Capstone)

Chapter 1: The “HTTPS Everywhere” Mandate

1.1 The Problem: The “Not Secure” Pain Point

We have successfully built our foundational “city”. We have a “Control Center” (DooD), “roads” (cicd-net), and “foundations” (our hybrid persistence model).

But we have a new, critical “pain point.” Our services can communicate, but they do so over unencrypted http. This is insecure, unprofessional, and does not emulate a real-world environment.

Pedagogical Failure (Browsers)

If we were to deploy GitLab right now on http://gitlab.cicd.local, our browsers would immediately betray the problem. We would be faced with a “Not Secure” warning in the address bar. This breaks user trust and, for many modern web features, can block functionality entirely.

If we tried to use a “magic” self-signed certificate (which we will deconstruct later), the problem gets worse. The browser would present a full-page, “Your connection is not private” interstitial error (like NET::ERR_CERT_AUTHORITY_INVALID or ERR_INSECURE_RESPONSE). This is a hard stop for most users.

Pedagogical Failure (Tools)

This “pain point” is not just cosmetic. It’s a functional “stack killer” for our specific set of tools.

  • curl: Any script using curl to access an HTTPS service with a bad certificate will fail with a hard error: curl: (60) SSL certificate problem: unable to get local issuer certificate. The only way around this is to use the --insecure flag, which is a terrible security practice.
  • The “Java Gotcha”: This is the most critical failure for our stack. Jenkins, Artifactory, and SonarQube are all Java applications. When our Jenkins container tries to make an API call or receive a webhook from our https://gitlab.cicd.local server, the Java Virtual Machine (JVM) will reject the untrusted certificate and throw a fatal javax.net.ssl.SSLHandshakeException.

This single Java exception will bring our entire automated pipeline to a grinding halt.

The Goal

Our goal is not just to “get rid of the warning.” It is to build a professional, secure, and trusted internal ecosystem. To do this, we must become our own “passport office.”

Chapter 2: A “First Principles” Guide to PKI and Trust

2.1 The Solution: Public Key Infrastructure (PKI)

To solve our “Not Secure” problem, we must adopt the same solution the entire internet uses: HTTPS.

HTTPS is not a single thing; it’s a combination of two: http (the protocol we already know) layered on top of SSL/TLS (Secure Sockets Layer/Transport Layer Security). This secure layer provides two fundamental guarantees:

  1. Encryption: It creates a secure, private “tunnel” between the client and the server. Any data sent through this tunnel (passwords, API keys, source code) is scrambled and unreadable to eavesdroppers.
  2. Identity: This is the more complex and crucial part. How does your browser know it’s talking to the real gitlab and not an imposter? The server must present a form of identification.

This system of identification is called Public Key Infrastructure (PKI).

The Analogy: “The Passport Office”

PKI works exactly like the international passport system.

  1. A Server Certificate is a “Passport”: It’s a digital file that proves a server’s identity (e.g., “I am gitlab“).
  2. A Certificate Authority (CA) is a “Passport Office”: It’s a trusted entity (like a government) that issues and signs the “passport,” attesting to its validity.
  3. A Trust Store is a “List of Trusted Governments”: Your browser and operating system come with a pre-installed “trust store” that contains a list of all the major, legitimate “passport offices” (like Let’s Encrypt, DigiCert, etc.) in the world.

When your browser connects to a server, it looks at the “passport” (the certificate) and checks the “signature” on it. If it was signed by a “passport office” (CA) in its trusted list, it grants access. If not, it blocks the connection.

2.2 Deconstructing Our Options

We need a “passport” (a certificate) for our services. We have three options for getting one.

Option 1: A Public CA (e.g., Let’s Encrypt)

This is the standard for all public websites. Let’s Encrypt is a globally trusted “passport office.” Your browser and OS already have it in their trust store.

Why we can’t use it: Let’s Encrypt must validate that you own a domain before issuing a certificate for it. It does this by connecting to your domain over the public internet. This process (Domain Validation or DV) is impossible for our local stack. The CA’s servers cannot reach localhost or our container’s hostname (gitlab). We cannot prove we “own” these private names, so the public CA will refuse to issue a certificate.

Option 2: A Simple Self-Signed Certificate

This is the “quick and dirty” method. A self-signed certificate is one that is signed by itself.

Why it fails: This is the equivalent of a “hand-drawn passport”. You are telling the browser, “I am gitlab, and I vouch for myself.” The browser, acting as the security guard, has no reason to trust you. It checks its list of “trusted passport offices” (the trust store) and finds no match. It correctly throws a full-page security warning (NET::ERR_CERT_AUTHORITY_INVALID) to protect you from a potential man-in-the-middle attack. This is the very SSLHandshakeException that will break our Java-based tools.

Option 3: Our Private/Local CA (The Solution)

This is the professional solution that balances our two problems. We will build our own “passport office”.

  1. We will become our own Certificate Authority (CA) by creating a Root CA certificate. This is our “passport office’s business license.”
  2. We will then use our new CA to issue “passports” (service certificates) for gitlab, jenkins, and all our other services.
  3. Finally, we will solve the “trust” problem by manually telling our local computers and containers: “This new ‘passport office’ is an official, trusted authority.”

From that point on, any passport (certificate) issued by our office will be automatically trusted by our local tools, browsers, and services.

Chapter 3: The “Hard Way” – Building Our CA with openssl

3.1 Why We Use openssl (The “First Principles”)

We will begin by creating our Certificate Authority using the openssl command-line tool directly. This is the “hard way.”

We are doing this for a specific pedagogical reason. We could use a simpler tool like easyrsa, but easyrsa is just a shell script wrapper that hides all the complexity. It automates all the tedious, “first principles” steps that a real CA must perform.

By using openssl directly, we are forced to deconstruct those steps. We will have to manually:

  1. Build the CA’s “Blueprint”: We will create an openssl.cnf configuration file that defines the rules and policies of our “passport office.”
  2. Create the “CA Database”: We will manually create the index.txt “master ledger” and the serial “ticket number dispenser” that openssl needs to track the certificates it issues.
  3. Manage the “Keys”: We will build the secure directory structure to protect our CA’s “master stamp” (its private key).

By doing this “hard way” first, you will understand what is actually happening under the hood. Only then will we introduce the “easy way” (easyrsa) as the automation tool it is.

We will build our entire PKI (Public Key Infrastructure) inside the ~/cicd_stack/ca directory we created on our host machine.

3.2 The “CA Blueprint”: openssl.cnf

Our first step is to create the “blueprint” for our new “passport office.” This is a configuration file, openssl.cnf, that tells openssl all of its rules. It defines the file locations, the default policies, and, most importantly, the extensions that make our root certificate a Certificate Authority.

Create this file on your host machine at ~/cicd_stack/ca/openssl.cnf.

[ ca ] default_ca = CA_default [ CA_default ] # --- Directory and file locations --- dir = ./pki certs = $dir/certs crl_dir = $dir/crl new_certs_dir = $dir/newcerts database = $dir/index.txt serial = $dir/serial RANDFILE = $dir/private/.rand # --- CA Key and Certificate --- private_key = $dir/private/ca.key certificate = $dir/certs/ca.pem # --- Policy & Extensions --- default_md = sha256 default_days = 3650 # 10 years policy = policy_strict x509_extensions = v3_ca [ policy_strict ] # Rules for signing: which fields must match the CA countryName = match stateOrProvinceName = match organizationName = match organizationalUnitName = optional commonName = supplied emailAddress = optional [ req ] # This section is used by the 'openssl req' command default_bits = 4096 distinguished_name = req_distinguished_name string_mask = utf8only default_md = sha256 # Tell 'openssl req -x509' to use our 'v3_ca' extensions x509_extensions = v3_ca [ req_distinguished_name ] # These are the DN fields it will prompt you for countryName = Country Name (2 letter code) stateOrProvinceName = State or Province Name localityName = Locality Name (eg, city) organizationName = Organization Name organizationalUnitName = Organizational Unit Name (eg, section) commonName = Common Name (e.g. server FQDN or YOUR name) emailAddress = Email Address [ v3_ca ] # --- Extensions for the Root CA certificate itself --- subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer basicConstraints = critical, CA:TRUE, pathlen:0 keyUsage = critical, digitalSignature, cRLSign, keyCertSign

Enter fullscreen mode

Exit fullscreen mode

Deconstruction

3.3 The “CA Database”

Now that we have our “blueprint” (openssl.cnf), we must create the physical “office” and “filing system” that openssl will use. This is the “database” that our CA will use to track every “passport” it ever issues.

This is the second “first principles” component that wrappers like easyrsa automate and hide from you. Our openssl.cnf file told openssl to look for these files in the ./pki directory; now, we must create them.

Action Plan: Creating the CA Directory

We will now create this entire structure using our first script, 01-create-ca.sh. This script will also generate our private key and our root certificate, completing our CA setup in one go.

Here is the code for 01-create-ca.sh.

#!/usr/bin/env bash # This script creates the CA directory structure, database, # and generates the Root CA private key and certificate. cd ~/cicd_stack/ca || (echo "no CICD directory present" && exit) if [ ! -f "openssl.cnf" ]; then echo "openssl.cnf not found in ~/cicd_stack/ca. Please create it before proceeding" exit fi # --- Configuration --- # The password for your new Root CA private key. # IMPORTANT: Change this to a strong, secure password. CA_PASSWORD="your_secure_password" # The Distinguished Name (DN) for your Root CA. # Customize these values for your organization. COUNTRY="ZA" STATE="Gauteng" LOCALITY="Johannesburg" ORG_NAME="Local CICD Stack" CA_CN="Local CICD Root CA" # --- Setup --- echo "--- Creating PKI directory structure ---" mkdir -p pki/{private,certs,newcerts,crl} # Set correct permissions for the "vault" chmod 700 pki/private echo "--- Creating CA database ---" touch pki/index.txt echo "1000" > pki/serial # --- Generate Root CA --- echo "--- 1. Generating Root CA Private Key (ca.key) ---" # Use -passout to provide the password non-interactively openssl genrsa -aes256 -passout pass:$CA_PASSWORD \ -out pki/private/ca.key 4096 # Secure the key chmod 400 pki/private/ca.key echo "--- 2. Generating Root CA Certificate (ca.pem) ---" # Use the '-subj' flag to pass DN info non-interactively openssl req -config openssl.cnf \ -key pki/private/ca.key \ -passin pass:$CA_PASSWORD \ -new -x509 -days 3650 -sha256 -extensions v3_ca \ -subj "/C=$COUNTRY/ST=$STATE/L=$LOCALITY/O=$ORG_NAME/CN=$CA_CN" \ -out pki/certs/ca.pem echo "--- CA creation complete ---" echo "CA Private Key: pki/private/ca.key" echo "CA Certificate: pki/certs/ca.pem"

Enter fullscreen mode

Exit fullscreen mode

Deconstruction

Let’s deconstruct the “Action” part of this script:

Action

Place the openssl.cnf file and the 01-create-ca.sh script in your ~/cicd_stack/ca directory on your host machine.

# (Run on HOST, inside ~/cicd_stack/ca) # 1. Make the script executable chmod +x 01-create-ca.sh # 2. Run the script ./01-create-ca.sh

Enter fullscreen mode

Exit fullscreen mode

Result:
The script will run non-interactively. When it finishes, you will have a fully functional Certificate Authority.

--- Creating PKI directory structure --- --- Creating CA database --- --- 1. Generating Root CA Private Key (ca.key) --- --- 2. Generating Root CA Certificate (ca.pem) --- --- CA creation complete --- CA Private Key: pki/private/ca.key CA Certificate: pki/certs/ca.pem

Enter fullscreen mode

Exit fullscreen mode

Chapter 4: Issuing Service Certificates (The SAN “Gotcha”)

4.1 The Goal: Creating Reusable “Passports”

We have successfully built our “passport office” (our Root CA). We have a secure “vault” for our ca.key and a public “business license” (ca.pem).

Now, we must start issuing “passports” (service certificates) for each of our 10 services. We could do this one by one, but that would be a tedious and error-prone manual process. Instead, we will immediately automate this.

We will create a single, reusable script named 02-issue-service-cert.sh. This script will be our “passport printer.” We will be able to run ./02-issue-service-cert.sh gitlab or ./02-issue-service-cert.sh jenkins.cicd.local, and it will automatically:

  1. Create a secure, isolated directory for the service.
  2. Generate a new private key for that service.
  3. Generate a Certificate Signing Request (CSR) with the correct “first principles” configuration.
  4. Use our Root CA to sign the request and issue a valid, trusted “passport.”

4.2 Action Plan: 02-issue-service-cert.sh

Here is the code for our “passport printer” script. Create this file on your host machine at ~/cicd_stack/ca/02-issue-service-cert.sh.

#!/usr/bin/env bash # This script issues a new, signed certificate for a service. # # USAGE: ./02-issue-service-cert.sh <service_name_or_path> # # EXAMPLE: ./02-issue-service-cert.sh gitlab # EXAMPLE: ./02-issue-service-cert.sh elk/elasticsearch # # This will create a new directory 'pki/services/gitlab' # containing 'gitlab.key.pem' and 'gitlab.crt.pem'. cd ~/cicd_stack/ca || (echo "no CICD directory present" && exit) # --- Configuration --- # This MUST match the password you used in '01-create-ca.sh' CA_PASSWORD="your_secure_password" # --- Script --- SERVICE_NAME=$1 CERT_NAME=$(basename "$SERVICE_NAME") if [ -z "$SERVICE_NAME" ]; then echo "ERROR: No service name provided." echo "USAGE: ./02-issue-service-cert.sh <service_name_or_path>" exit 1 fi echo "--- Preparing environment for service: $SERVICE_NAME ---" echo "--- Certificate simple name will be: $CERT_NAME ---" # 1. Define all file paths SERVICE_DIR="pki/services/$SERVICE_NAME" KEY_FILE="$SERVICE_DIR/$CERT_NAME.key.pem" CSR_FILE="$SERVICE_DIR/$CERT_NAME.csr" CERT_FILE="$SERVICE_DIR/$CERT_NAME.crt.pem" EXT_FILE="$SERVICE_DIR/v3.ext" # 2. Create the service's private directory mkdir -p $SERVICE_DIR chmod 700 $SERVICE_DIR # 3. Generate the service's Private Key echo "--- 1. Generating Private Key ($CERT_NAME.key.pem) ---" openssl genrsa -out $KEY_FILE 4096 chmod 400 $KEY_FILE # 4. Create the SAN (Subject Alternative Name) config file # This is critical for modern browser compatibility. echo "--- 2. Creating SAN config file (v3.ext) ---" cat > $EXT_FILE <<- EOM [ v3_ext ] subjectAltName = @alt_names [alt_names] DNS.1 = $CERT_NAME DNS.2 = localhost IP.1 = 127.0.0.1 EOM # 5. Generate a Certificate Signing Request (CSR) echo "--- 3. Generating Certificate Signing Request ($CERT_NAME.csr) ---" openssl req -new -sha256 \ -key $KEY_FILE \ -subj "/C=ZA/ST=Gauteng/L=Johannesburg/O=Local CICD Stack/CN=$CERT_NAME" \ -out $CSR_FILE # 6. Sign the CSR with our Root CA echo "--- 4. Signing the certificate with our Root CA ---" openssl ca -config openssl.cnf \ -extensions v3_ext -extfile $EXT_FILE \ -days 365 -notext -md sha256 \ -passin pass:$CA_PASSWORD \ -in $CSR_FILE \ -out $CERT_FILE \ -batch # Use non-interactive "batch" mode # 7. Clean up temporary files rm $CSR_FILE rm $EXT_FILE echo "--- Successfully issued certificate for $SERVICE_NAME ---" echo "Certificate: $CERT_FILE" echo "Verifying SANs..." # 8. Verify the certificate's SANs openssl x509 -in $CERT_FILE -noout -text | grep -A1 "Subject Alternative Name"

Enter fullscreen mode

Exit fullscreen mode

4.3 Deconstruction (The SAN “Gotcha”)

This script is the heart of our PKI, and it contains two critical, “first principles” lessons that we discovered during our testing.

Lesson 1: The “SAN Gotcha” (Why CN is Deprecated)

The most important part of this script is Step 4. We don’t just generate a CSR; we first create a temporary config file called v3.ext:

# (Inside 02-issue-service-cert.sh) cat > $EXT_FILE <<- EOM [ v3_ext ] subjectAltName = @alt_names [alt_names] DNS.1 = $CERT_NAME DNS.2 = localhost IP.1 = 127.0.0.1 EOM

Enter fullscreen mode

Exit fullscreen mode

This is the Subject Alternative Name (SAN) field. In the early days of the internet, browsers only checked the Common Name (CN) (which we set in Step 5 with -subj "/CN=...").

However, this CN field was ambiguous and is now deprecated. All modern browsers and tools (including Java) ignore the CN field and only validate the SAN list.

Our script fixes this by:

  1. Creating a v3.ext file that defines the SAN extension (subjectAltName = @alt_names).
  2. Populating the [alt_names] list with all the names this certificate should be valid for.
  3. Telling openssl ca to use these extensions with the -extensions v3_ext -extfile $EXT_FILE flags.

We include three entries for maximum compatibility:

Lesson 2: The “Nested Path Gotcha” (Using basename)

The second “pain point” we discovered was when we tried to issue a certificate for elk/elasticsearch.cicd.local. Our script failed because it tried to create a file named .../elk/elasticsearch.cicd.local/elk/elasticsearch.cicd.local.key.pem, which is redundant and wrong.

The solution is the basename command:

# (Inside 02-issue-service-cert.sh) SERVICE_NAME=$1 CERT_NAME=$(basename "$SERVICE_NAME")

Enter fullscreen mode

Exit fullscreen mode

This command is a standard Linux utility that strips the directory path from a string.

This allows us to use the full $SERVICE_NAME to create the directory path (pki/services/elk/elasticsearch.cicd.local) but use the clean $CERT_NAME for all the filenames (e.g., elasticsearch.cicd.local.key.pem).

Lesson 3: The openssl ca Command

The final command, openssl ca, is the “passport office” at work.

# (Inside 02-issue-service-cert.sh) openssl ca -config openssl.cnf \ -extensions v3_ext -extfile $EXT_FILE \ -days 365 -notext -md sha256 \ -passin pass:$CA_PASSWORD \ -in $CSR_FILE \ -out $CERT_FILE \ -batch

Enter fullscreen mode

Exit fullscreen mode

4.4 Action Plan: 03-issue-all-certs.sh

Now that we have our powerful, reusable “passport printer” script, we can create a simple automation script to issue “passports” for all 10 of our services.

Create this file on your host machine at ~/cicd_stack/ca/03-issue-all-certs.sh.

#!/usr/bin/env bash # This script automates the issuance of certificates for all # services in our 10-article CI/CD stack by looping # over an array and calling our '02-issue-service-cert.sh' script. # --- Configuration --- # This array defines the service names. These names will be used # for the certificate Common Name (CN) and the directory path # inside 'pki/services/'. # # Note the nested paths for ELK, which our script handles automatically. SERVICES=( "gitlab.cicd.local" "jenkins.cicd.local" "artifactory.cicd.local" "sonarqube.cicd.local" "mattermost.cicd.local" "elk/elasticsearch.cicd.local" "elk/logstash.cicd.local" "elk/kibana.cicd.local" "prometheus.cicd.local" "grafana.cicd.local" ) ISSUER_SCRIPT="./02-issue-service-cert.sh" # --- Pre-run Checks --- if [ ! -f "$ISSUER_SCRIPT" ]; then echo "ERROR: Issuer script '$ISSUER_SCRIPT' not found." echo "Please ensure you are in the '~/cicd_stack/ca' directory." exit 1 fi if [ ! -x "$ISSUER_SCRIPT" ]; then echo "ERROR: Issuer script '$ISSUER_SCRIPT' is not executable." echo "Please run: chmod +x $ISSUER_SCRIPT" exit 1 fi echo "---=== Issuing all service certificates ===---" echo "This will use the CA password hardcoded in '$ISSUER_SCRIPT'." # --- Main Loop --- for service in "${SERVICES[@]}"; do echo "" echo "------------------------------------------------" echo "Issuing certificate for: $service" echo "------------------------------------------------" # Call the issuer script with the service name ./$ISSUER_SCRIPT "$service" if [ $? -ne 0 ]; then echo "ERROR: Failed to issue certificate for $service. Aborting." exit 1 fi done echo "" echo "------------------------------------------------" echo "---=== All service certificates issued successfully ===---" echo "You can view the new directory structure:" ls -lR ~/cicd_stack/ca/pki/services/

Enter fullscreen mode

Exit fullscreen mode

Action

From your host machine (inside ~/cicd_stack/ca), make both scripts executable, then run the automation script.

# (Run on HOST, inside ~/cicd_stack/ca) chmod +x 02-issue-service-cert.sh chmod +x 03-issue-all-certs.sh ./03-issue-all-certs.sh

Enter fullscreen mode

Exit fullscreen mode

Verification:
The script will loop through all 10 services. The final ls -lR command will display the complete, organized directory structure you’ve created, with each service’s key and certificate in its own isolated folder (e.g., pki/services/gitlab/, pki/services/elk/elasticsearch/, etc.).

Chapter 5: The New “Pain Point” – Establishing Trust

5.1 The “Failure First”: The “Untrusted Issuer”

We have successfully built our “passport office” and printed a full set of valid “passports” for all our services. Our pki/services directory is full of certificates that are cryptographically perfect, have a valid date range, and contain the correct Subject Alternative Names (SANs).

This leads us to a new, critical “pain point.”

If we were to deploy a service right now using our new gitlab.cicd.local.key.pem certificate, our browsers would still show a full-page security warning.

However, the error would be different. It would no longer be NET::ERR_CERT_COMMON_NAME_INVALID (which our SANs fixed). Instead, it would be NET::ERR_CERT_AUTHORITY_INVALID.

The Analogy: The “Untrusted Passport Office”

This new error is a crucial distinction. The browser is no longer complaining about the “passport” itself (gitlab.cicd.local.crt.pem). It’s complaining about the issuer.

The browser is acting as the border agent, looking at our valid “passport” and saying:

“This passport looks perfectly filled out. The photo matches, the name is correct, and it’s not expired. But I’ve never heard of the ‘passport office’ (our ‘Local CICD Root CA’) that issued this. For all I know, you built that office in your garage. I don’t trust it.”

We have created a valid certificate from an untrusted authority. Our next and final step is to establish that trust.

5.2 The “Disjointed Trust Stores” Principle

To solve this new “untrusted” pain point, we must understand a critical, “first principles” concept: there is no single “trust” button on a modern operating system.

When we ran our 04-trust-ca-host.sh script, we discovered that curl was fixed, but Chrome and Firefox were not. This is because our “government” (the host OS) is fragmented, and different applications pledge allegiance to different “lists of trusted passport offices” (Trust Stores).

To fix this, we must manually add our ca-cert.pem to several independent trust stores:

  1. The OpenSSL System Trust Store: This is the main list for the operating system, located at /etc/ssl/certs/. It is used by most command-line tools like curl, wget, git, and apt. We already fixed this with sudo update-ca-certificates.
  2. The Firefox NSS Trust Store: Firefox, by design, ignores the system trust store. It maintains its own private, isolated database (a file named cert9.db) inside your user’s profile directory.
  3. The System-wide NSS Trust Store: Google Chrome also uses the NSS (Network Security Services) database, but it looks at a different, system-wide one (often ~/.pki/nssdb/). This is why fixing the OS store didn’t fix Chrome.
  4. The “Java Gotcha” (cacerts): This is the most important “gotcha” for our stack. The Java Virtual Machine (JVM) ignores all system trust stores. Java has its own separate, single-file database called cacerts. When we deploy Jenkins, Artifactory, and SonarQube, we must manually import our CA into this specific file, or they will all fail with SSLHandshakeException.

Our “Action Plan” must therefore be a multi-pronged attack, targeting each of these trust stores individually.

5.3 Action Plan: Trusting Our CA on the Host

Now that we understand the problem, we will execute the solution. Our script 04-trust-ca-host.sh is a precision tool designed to inject our ca.pem file into all three “disjointed trust stores” on our host machine.

Here is the code for 04-trust-ca-host.sh.

#!/usr/bin/env bash # This script trusts our Root CA on the host machine. # It will prompt for your password for 'sudo' commands. # # It trusts the CA in THREE places: # 1. The Host OS (for curl, apt, git, etc.) # 2. The system-wide NSS database (for Chrome/Chromium) # 3. The Firefox browser (which uses its own private store) echo "--- Navigating to CA directory ---" # Navigate to the CA directory to find the certs cd ~/cicd_stack/ca || (echo "ERROR: ~/cicd_stack/ca directory not found." && exit) CA_CERT_NAME="cicd-stack-ca.crt" CA_CERT_PATH="pki/certs/ca.pem" CA_NICKNAME="Local CICD Root CA" if [ ! -f "$CA_CERT_PATH" ]; then echo "ERROR: Root CA certificate not found at $CA_CERT_PATH" exit 1 fi # --- 0. Install Prerequisites (NSS Tools) --- echo "--- Installing 'libnss3-tools' (for certutil) ---" sudo apt-get update -qq && sudo apt-get install -y -qq libnss3-tools # --- 1. Trust CA in Host OS (for curl, Chrome, ...) --- echo "--- 1. Trusting CA in Host OS (OpenSSL store) ---" sudo cp $CA_CERT_PATH /usr/local/share/ca-certificates/$CA_CERT_NAME sudo update-ca-certificates # --- 2. Trust CA in System-wide NSS DB (for Chrome) --- echo "--- 2. Trusting CA in NSS DB (for Chrome) ---" # Create the NSS database directory if it doesn't exist mkdir -p $HOME/.pki/nssdb # Add the certificate to the NSS database # -d sql:$HOME/.pki/nssdb : Specifies the database path # -A : Add a certificate # -n : Nickname # -t "C,T,C" : Set trust attributes (C=SSL Client, T=SSL Server, C=Email) certutil -A -n "$CA_NICKNAME" -t "C,T,C" -i $CA_CERT_PATH -d sql:$HOME/.pki/nssdb # --- 3. Trusting CA in Firefox --- echo "--- 3. Trusting CA in Firefox (private store) ---" # Find all Firefox profile databases (cert9.db) FF_PROFILES_DB=$(find $HOME/.mozilla/firefox -name "cert9.db") if [ -z "$FF_PROFILES_DB" ]; then echo "WARNING: Could not find Firefox cert9.db. Skipping Firefox trust." echo "You may need to add the CA manually via Firefox settings." else for db in $FF_PROFILES_DB; do PROFILE_PATH=$(dirname $db) echo "--- Found Firefox profile, adding CA to: $PROFILE_PATH ---" # Add the CA to the Firefox NSS database certutil -A -n "$CA_NICKNAME" -t "C,T,C" -i $CA_CERT_PATH -d sql:$PROFILE_PATH done fi echo "--- Host trust setup complete ---" echo "IMPORTANT: Please restart Google Chrome for changes to take effect."

Enter fullscreen mode

Exit fullscreen mode

Deconstruction

This script is a perfect example of our “first principles” approach, as it surgically targets each trust store:

Action

From your host machine (inside your 0006_cicd_part02_certificate_authority directory), make the script executable and run it.

# (Run on HOST) chmod +x 04-trust-ca-host.sh ./04-trust-ca-host.sh

Enter fullscreen mode

Exit fullscreen mode

Result:
The script will prompt for your sudo password, then proceed to install the tools and update all three trust stores. You will see output like “1 added, 0 removed; done.” from update-ca-certificates and confirmation from certutil.

5.4 Action Plan: Trusting the “Control Center” (dev-container)

We have successfully taught our host machine to trust our CA. Now we must do the same for our dev-container.

This is a critical step. Our dev-container is our “Control Center.” Its tools (curl, git, and most importantly, our future Python API scripts) must be able to make secure HTTPS connections to our new services. If our container doesn’t trust our CA, our entire automation plan will fail with the same SSLHandshakeException and certificate problem errors.

The “No-Rebuild” Solution

We will use the flexible architecture we built in Article 1 to solve this without rebuilding our container image.

  1. The data/ Mount: Our dev-container.sh script mounts our host’s data/ directory into the container.
  2. The entrypoint.sh Mount: Our dev-container.sh script also mounts our host’s entrypoint.sh script, which runs every time the container starts.

We will copy our public CA certificate into the host’s data/ directory. Then, we will edit our entrypoint.sh script to automatically install this certificate at startup.

Action 1: Copy the CA Certificate

From your host machine, copy your public “business license” (ca.pem) into the data/ directory of your FromFirstPrinciples project.

# (Run on HOST, from your 0006_cicd... directory) # This copies our CA cert into the shared data folder cp ~/cicd_stack/ca/pki/certs/ca.pem ~/Documents/FromFirstPrinciples/data/ca-cert.pem

Enter fullscreen mode

Exit fullscreen mode

Action 2: Edit entrypoint.sh

Now, open your entrypoint.sh file (in your FromFirstPrinciples directory) with your text editor. Add the following block of code directly after the sudo service ssh restart line.

#!/bin/sh sudo service ssh restart # --- ADD THIS BLOCK TO TRUST THE LOCAL CA --- CA_CERT_PATH="data/ca-cert.pem" if [ -f "$CA_CERT_PATH" ]; then echo "--- Found Local CA certificate, installing to system trust store ---" # Copy the CA cert into the system's trust store sudo cp "$CA_CERT_PATH" /usr/local/share/ca-certificates/cicd-stack-ca.crt # Update the system's CA list sudo update-ca-certificates else echo "--- No Local CA certificate found at $CA_CERT_PATH, skipping system trust ---" fi # --- END OF BLOCK --- # Check for GPG key and gitconfig on the persistent data volume if [ -e data/private.pgp ]; then gpg --import data/private.pgp fi ...

Enter fullscreen mode

Exit fullscreen mode

(The rest of your entrypoint.sh file remains the same.)

Deconstruction

This logic is simple and powerful:

  1. It checks if the data/ca-cert.pem file (which we just copied) is present.
  2. If it is, it copies it into the container’s system trust directory (/usr/local/share/ca-certificates/).
  3. It runs sudo update-ca-certificates inside the container to make its system tools (like curl and git) trust our CA.

Action 3: Recreate the dev-container

Because we just changed the entrypoint.sh script (which is bind-mounted), we don’t need to rebuild the image, but we do need to recreate the container to run the new script.

As we’ve noted, your entrypoint.sh script also handles your GPG key, which requires an interactive prompt. Therefore, we must use our docker stop/rm and ./dev-container.sh flow.

From your host machine (in the FromFirstPrinciples directory):

# (Run on HOST) # 1. Stop and remove the old container docker stop dev-container && docker rm dev-container # 2. Run the new container (which will use the modified entrypoint) ./dev-container.sh

Enter fullscreen mode

Exit fullscreen mode

Result:
As the container starts, you will see our new messages in the log output, culminating in:

--- Found Local CA certificate, installing to system trust store --- Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done.

Enter fullscreen mode

Exit fullscreen mode

This confirms our “Control Center” now also trusts our CA.

5.5 Verification: The “Success Second” (Python HTTPS Server)

We have now executed three complex, “first principles” operations:

  1. We created a Root CA and a full set of service certificates, solving the “SAN Gotcha” and “Nested Path Gotcha.”
  2. We manually configured our Host OS to trust our CA, fixing curl, Chrome, and Firefox.
  3. We configured our dev-container to trust our CA by modifying its entrypoint.sh.

The plan is complete, but we must verify it. We will now run a simple, local HTTPS server on our host machine using one of the “passports” we just printed (our gitlab.cicd.local certificate). We will then test if our browsers and our dev-container trust it.

Action 1: The Python Test Server

Here is the code for 05-verify-trust.py. This script is a simple, standard-library-only HTTPS server.

Create this file on your host machine at ~/Documents/FromFirstPrinciples/articles/0006_cicd_part02_certificate_authority/05-verify-trust.py.

#!/usr/bin/env python3 import http.server import ssl import os from pathlib import Path # --- Configuration --- # This is the "simple name" of the certificate we want to use. # It must match one of the subdirectories we created in pki/services/ # We will use 'gitlab' as our test certificate. CERT_NAME = "gitlab.cicd.local" # Find the '~/cicd_stack/ca' directory from the user's home HOME_DIR = Path.home() CA_DIR = HOME_DIR / "cicd_stack" / "ca" PKI_DIR = CA_DIR / "pki" CERT_DIR = PKI_DIR / "services" / CERT_NAME # Paths to the certificate and private key files CERT_FILE = CERT_DIR / f"{CERT_NAME}.crt.pem" KEY_FILE = CERT_DIR / f"{CERT_NAME}.key.pem" # Server configuration HOST = "localhost" PORT = 8443 # --- Validation --- if not PKI_DIR.exists(): print(f"--- ERROR ---") print(f"Could not find CA directory at: {CA_DIR}") print("Please ensure you have run '01-create-ca.sh' successfully.") exit(1) if not CERT_FILE.exists() or not KEY_FILE.exists(): print(f"--- ERROR ---") print(f"Could not find certificate files for '{CERT_NAME}'.") print(f"Expected cert at: {CERT_FILE}") print(f"Expected key at: {KEY_FILE}") print("Please run './03-issue-all-certs.sh' first.") exit(1) # --- Server Setup --- print(f"--- Starting HTTPS server on https://{HOST}:{PORT} ---") print(f"Loading certificate: {CERT_FILE}") print(f"Loading private key: {KEY_FILE}") # Create a standard TCP server httpd = http.server.HTTPServer( (HOST, PORT), http.server.SimpleHTTPRequestHandler ) # Create an SSL context context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) context.load_cert_chain(certfile=CERT_FILE, keyfile=KEY_FILE) # Wrap the server socket with our new SSL context httpd.socket = context.wrap_socket(httpd.socket, server_side=True) print(f"--- Server is running. Open https://{HOST}:{PORT} in your browser. ---") print("Press Ctrl+C to stop.") try: httpd.serve_forever() except KeyboardInterrupt: print("\n--- Server stopped. ---")

Enter fullscreen mode

Exit fullscreen mode

Deconstruction

Action 2: Run the Server and Verify

From your host machine (inside your 0006_cicd_part02_certificate_authority directory), make the script executable and run it:

# (Run on HOST) chmod +x 05-verify-trust.py ./05-verify-trust.py

Enter fullscreen mode

Exit fullscreen mode

The server is now running on https://localhost:8443.

Verification Step 1: Browsers (Host)

  1. Open Google Chrome (in an Incognito window to be safe) and go to https://localhost:8443.
  2. Open Firefox and go to https://localhost:8443.

Result: Both browsers should show a secure padlock icon 🔒. They will not show a “Your connection is not private” warning. This proves that our 04-trust-ca-host.sh script worked and successfully configured all three trust stores.

Verification Step 2: curl (Host)

Open a new host terminal (leaving the server running) and test with curl using both the name and the IP address.

# (Run on HOST) # Test 1: By hostname curl https://localhost:8443 # Test 2: By IP address curl https://127.0.0.1:8443

Enter fullscreen mode

Exit fullscreen mode

Result: Both commands will succeed and print the HTML for a directory listing. They will not fail with an “SSL certificate problem.” This proves our host’s OpenSSL trust store is fixed and our SAN IP:127.0.0.1 entry is working.

Verification Step 3: curl (Container)

This is the final and most important test. We must prove that our dev-container—our “Control Center”—also trusts our new CA. We will do this by running our Python test server inside the container and then, from a second container shell, trying to curl it.

This is a perfect, hermetic test of the container’s internal trust store.

Action 1 (Host): Copy Certs into the “Dropbox”
Our test server, 05-verify-trust.py, is configured to find certificates at ~/cicd_stack/ca/pki/services/.... This path doesn’t exist inside our container, so we must first get the gitlab certificate folder into the container. We will use our mounted data/ directory as the “dropbox.”

From your host machine:

# (Run on HOST) # Copy the gitlab certs/key folder into the 'data' directory cp -r ~/cicd_stack/ca/pki/services/gitlab.cicd.local/ ~/Documents/FromFirstPrinciples/data/

Enter fullscreen mode

Exit fullscreen mode

Action 2 (Container): Run the Server
Now, we’ll enter our dev-container and set up the test. This requires two terminals.

Terminal 1 (dev-container shell 1):

# (Run on HOST) docker exec -it dev-container bash # (Inside dev-container - Terminal 1) # 1. Re-create the path structure the script expects mkdir -p ~/cicd_stack/ca/pki/services/ # 2. Copy the certs from our 'dropbox' to the expected path cp -r ~/data/gitlab.cicd.local/ ~/cicd_stack/ca/pki/services/ # 3. Navigate to the article directory cd ~/articles/0006_cicd_part02_certificate_authority/ # 4. Run the Python server python3 05-verify-trust.py

Enter fullscreen mode

Exit fullscreen mode

Result:
The server will start successfully inside the container, as it can now find its certificates.

--- Starting HTTPS server on https://localhost:8443 --- Loading certificate: /home/warren_jitsing/cicd_stack/ca/pki/services/gitlab.cicd.local/gitlab.cicd.local.crt.pem Loading private key: /home/warren_jitsing/cicd_stack/ca/pki/services/gitlab.cicd.local/gitlab.cicd.local.key.pem --- Server is running. Open https://localhost:8443 in your browser. --- Press Ctrl+C to stop.

Enter fullscreen mode

Exit fullscreen mode

Terminal 2 (dev-container shell 2):
While the server is running in the first terminal, open a second host terminal and get another shell inside the dev-container.

# (Run on HOST - Terminal 2) docker exec -it dev-container bash

Enter fullscreen mode

Exit fullscreen mode

Now, from this second shell, we will run curl to connect to the server running in our first shell.

# (Inside dev-container - Terminal 2) # 5. Test connection to localhost curl https://localhost:8443 # 6. Test connection to 127.0.0.1 curl https://127.0.0.1:8443

Enter fullscreen mode

Exit fullscreen mode

Result:
Both commands will succeed, returning the HTML of the directory listing.

Explanation:
This is the ultimate verification. curl (running inside the container) is using the container’s system trust store. This test proves that:

  1. Our entrypoint.sh script successfully ran at startup.
  2. It correctly found data/ca-cert.pem and ran sudo update-ca-certificates.
  3. The container’s trust store is now fixed, and all tools inside our “Control Center” (like curl, git, and our future Python scripts) will fully trust our new CA.

Cleanup (Terminal 1):
You can now stop the Python server with Ctrl+C and remove the temporary certificate directory.

# (Inside dev-container - Terminal 1) rm -rf ~/cicd_stack/ rm -rf ~/data/gitlab.cicd.local/

Enter fullscreen mode

Exit fullscreen mode

Chapter 6: The “Easy Way” – Automating with easyrsa

6.1 The “Why”: Appreciating Automation

In the previous chapters, we built a fully functional, secure Certificate Authority “the hard way.” We meticulously crafted our openssl.cnf, built our PKI directory structure, and manually managed our “ledger” (index.txt) and “ticket dispenser” (serial). We deconstructed the complex openssl ca command and learned, through failure, that we must manually build and pass a v3.ext file to handle the critical Subject Alternative Name (SAN) requirement.

This was an essential “first principles” exercise. Now that we have done the hard work and understand every moving part, we can appreciate the “easy button.”

We will now introduce easyrsa, a tool whose sole purpose is to automate every tedious step we just learned.

6.2 The “What”: easyrsa is a Wrapper

easyrsa is not a replacement for openssl. It is a shell script wrapper around openssl. It was developed by the OpenVPN project to simplify the creation of a PKI.

It is our “automated passport office clerk.” It takes our simple, high-level commands (like “make a CA” or “make a server certificate”) and, behind the scenes, runs the same complex openssl commands we just did, automatically managing the index.txt, serial file, and configuration templates for us.

We will now run a self-contained demo to prove this.

6.3 Action Plan: 06-run-easyrsa-demo.sh

Here is the code for 06-run-easyrsa-demo.sh. This script will install easyrsa by cloning its Git repository (as it is not available in the default Debian repositories), and then use it to build a second, separate “demo” CA.

Create this file on your host machine at ~/Documents/FromFirstPrinciples/articles/0006_cicd_part02_certificate_authority/06-run-easyrsa-demo.sh.

#!/usr/bin/env bash # This script demonstrates the "easy way" to create a CA # and issue a certificate using the 'easyrsa' wrapper. # # It is for demonstration only and does not use our # main '~/cicd_stack/ca' directory. DEMO_DIR="easyrsa-demo" REPO_URL="https://github.com/OpenVPN/easy-rsa.git" REPO_BRANCH="v3.2.4" SERVICE_NAME="gitlab-demo" # --- 1. Installation (via Git) --- echo "--- 1. Installing 'git' (prerequisite) ---" sudo apt-get update -qq && sudo apt-get install -y -qq git # --- 2. Setup --- echo "--- 2. Cloning easyrsa branch '$REPO_BRANCH' into demo directory: $DEMO_DIR ---" rm -rf $DEMO_DIR git clone --depth=1 -b $REPO_BRANCH $REPO_URL $DEMO_DIR # Navigate into the new demo directory cd $DEMO_DIR || (echo "Failed to clone easyrsa. Aborting." && exit 1) # Navigate into the easyrsa3 directory where the executable is cd easyrsa3 || (echo "Failed to find 'easyrsa3' directory. Aborting." && exit 1) # --- 3. Create the PKI and Root CA --- echo "--- 3. Initializing PKI and building Root CA ---" # Initialize the PKI (creates 'pki' dir, index, serial, etc.) ./easyrsa init-pki # Build the CA, 'nopass' = no password on the private key # 'batch' = use defaults for the DN (Country, Org, etc.) ./easyrsa --batch build-ca nopass # --- 4. Issue a Service Certificate --- echo "--- 4. Issuing service certificate for '$SERVICE_NAME' (with SANs) ---" # We MUST explicitly pass the SANs. We will add the service name, # localhost, and 127.0.0.1, just like our "hard way" script. SANS="DNS:$SERVICE_NAME,DNS:localhost,IP:127.0.0.1" echo "--- Using SANs: $SANS ---" ./easyrsa --batch --san="$SANS" build-server-full $SERVICE_NAME nopass # --- 5. Verification --- echo "--- Demo complete. Verifying generated files: ---" echo "1. CA Certificate:" ls -l pki/ca.crt echo "2. Service Private Key:" ls -l pki/private/$SERVICE_NAME.key echo "3. Service Certificate:" ls -l pki/issued/$SERVICE_NAME.crt echo "" echo "--- Verifying SANs in the 'easyrsa' generated cert: ---" openssl x509 -in pki/issued/$SERVICE_NAME.crt -noout -text | grep -A1 "Subject Alternative Name" # Clean up by returning to the original directory cd ../..

Enter fullscreen mode

Exit fullscreen mode

Deconstruction

This script automates the entire process, perfectly illustrating why openssl is so complex.

Action

From your host machine (inside your 0006_cicd_part02_certificate_authority directory), make this new script executable and run it.

# (Run on HOST) chmod +x 06-run-easyrsa-demo.sh ./06-run-easyrsa-demo.sh

Enter fullscreen mode

Exit fullscreen mode

Result:
The script will install git, clone the easyrsa repository, and then run the simplified workflow. The final openssl command will print the SAN field, proving that easyrsa is just an automated wrapper around the same “first principles” we learned.

Chapter 7: Conclusion and Next Steps

7.1 What We’ve Built

We have successfully built a complete, local Public Key Infrastructure (PKI) from “first principles.” We started with the “pain point” of insecure http and browser errors and deconstructed the entire chain of trust.

We rejected the “magic” of simple, self-signed certificates and instead built our own “passport office”—a full Certificate Authority. We did it the “hard way” with openssl to understand every moving part:

We then solved the “new pain point” of trust by deconstructing the “disjointed trust stores” on our host. We surgically added our CA to the OpenSSL store (for curl), the NSS store (for Chrome), and the Firefox store (for Firefox) using our 04-trust-ca-host.sh script.

Finally, we verified our entire setup end-to-end, proving that both our host machine and our dev-container now fully trust our new “passport office.”

7.2 Next Steps

Our “city” is finally ready.

We are finally ready to build our first “skyscraper.” In the next article, we will deploy GitLab, the “Central Library” for our source code. We will use the docker run command to deploy it onto our foundation and use the gitlab.crt.pem and gitlab.key.pem files we just created to make it fully secure with HTTPS from the moment it starts.

For further actions, you may consider blocking this person and/or reporting abuse

Залишити відповідь

Ваша e-mail адреса не оприлюднюватиметься. Обов’язкові поля позначені *

Читайте також