End-to-End Nodejs | Postgres | AWS DevOps Project

This repository contains scripts and Kubernetes manifests for deploying the Nodejs application with Postgres Database on an AWS EKS cluster with an accompanying ECR repository and EBS volumes. The deployment includes setting up an Ingress controller, monitoring with Prometheus and Grafana, and a continuous deployment pipeline.

• AWS CLI configured with appropriate permissions
• Docker installed and configured
• kubectl installed and configured to interact with your Kubernetes cluster
• Terraform installed
• Helm installed
• GitHub_CLI installed
• K9s installed
• Beekeeper-Studio For Database Access

The build.sh script contains a set of variables that you need to customize according to your AWS environment and deployment requirements. Here’s how you can update them:

1. Open the build.sh script in a text editor of your choice.

2. Update the variables at the top of the script with your specific configurations:

   cluster_name="YOUR_EKS_CLUSTER_NAME"
   region="YOUR_AWS_REGION"
   aws_id="YOUR_AWS_ACCOUNT_ID"
   repo_name="YOUR_REPO_NAME"
   domain="YOUR_DOMAIN.com"
   dbsecret="POSTGRES_K8S_SECRET"
   namespace="nodejs-app"

    

3. Save the changes to the build.sh script.

• cluster_name: This is the name of your Amazon EKS cluster.
• region: This is the AWS region where your resources are located.
• aws_id: Your AWS account ID which is required to construct the ECR image name.
• domain: The domain that your application will use.
• dbsecret: If you update this variable make sure to update it in k8s/database.yaml in the Postgres environment variable name: db-password-secret
• namespace: The Kubernetes namespace where your application will be deployed. If you change the namespace, ensure to also update it in your Kubernetes manifests located in the k8s directory.

After updating these variables, the script will use them to deploy your application, ensuring that the correct resources are targeted and that your application is accessible via your specified domain.

Reminder: Always double-check the variable values to match your AWS setup and application requirements before running the build.sh script.

1. Navigate to terraform/monitoring.tf
2. Update applications host with your domain
3. Do the same in k8s/app.yaml

    grafana:
      adminUser: admin
      adminPassword: admin
      enabled: true
      ingress:
        enabled: true
        ingressClassName: nginx
        annotations:
          cert-manager.io/cluster-issuer: letsencrypt-production
        hosts:
          - grafana.[YOUR_DOMAIN]
        tls:
          - secretName: grafana-tls
            hosts:
              - grafana.[YOUR_DOMAIN]

IMPORTANT: Ensure you have updated the domain for all the services, Grafana, Alertmanager and Prometheus

The build.sh script automates the process of setting up the infrastructure on AWS, building a Docker image for the Go Survey app, and deploying it to the Kubernetes cluster. Here’s a step-by-step explanation of what the script does:

1. Creating Infrastructure: Sets up the AWS EKS cluster, ECR repository, and EBS volumes using Terraform.

2. Update Kubeconfig: Configures kubectl to interact with the newly created EKS cluster.

3. Build Docker Image: Removes any existing Docker images and builds a new one with the Go Survey app.

4. Push Docker Image: Logs into ECR and pushes the new Docker image to the repository.

5. Generate DB Password: Generates a random password for the Postgres database.

6. K8s Secret: Creates a secret key for the Postgres database from the previously generated password.

7. Deploy to Kubernetes: Creates the specified namespace if it doesn’t exist and applies the Kubernetes manifests from the k8s directory.

8. Wait for Deployment: Waits for 60 seconds to allow the Kubernetes resources to be deployed.

9. Ingress URL: Retrieves the Ingress URL for accessing the deployed application.

10. Application URLs: Prints out the URLs for accessing the Go Survey app and the monitoring tools (Alertmanager, Prometheus, Grafana).

1. Clone the repository and navigate to the root directory.

2. Make the deployment script executable:

   chmod +x build.sh

    

3. Run the deployment script:

   ./build.sh

    

4. Follow the post-deployment steps printed by the script to update your DNS records with the provided Ingress URL.

Once the script has completed, you will need to add DNS records to point your domain to the services deployed. Follow the instructions outputted by the script to set up CNAME records for your application.

1. Access the Zone Editor in the Cpanel of your domain

   

2. Next to your domain Add CNAME Record

3. In the NAME add YOUR_APP_NAME.YOUR_DOMAIN and add the INGRESS_URL in CNAME

Repeat the same steps for the rest of the services

After DNS configuration, you should be able to access the following services:

• Nodejs App: http://nodeapp.[your-domain]
• Alertmanager: http://alertmanager.[your-domain]
• Prometheus: http://prometheus.[your-domain]
• Grafana: http://grafana.[your-domain]

Replace [your-domain] with your actual domain name.

This project is equipped with GitHub Actions workflows to automate the Continuous Integration (CI) and Continuous Deployment (CD) processes.

The CI workflow is triggered on pushes to the main branch. It performs the following tasks:

• Checks out the code from the repository.
• Configures AWS credentials using secrets stored in the GitHub repository.
• Logs in to Amazon ECR.
• Builds the Docker image for the Nodejs app.
• Tags the image and pushes it to the Amazon ECR repository.

The CD workflow is triggered upon the successful completion of the CI workflow. It performs the following tasks:

• Checks out the code from the repository.
• Configures AWS credentials using secrets stored in the GitHub repository.
• Sets up kubectl with the required Kubernetes version.
• Deploys the Kubernetes manifests found in the k8s directory to the EKS cluster.

The following secrets need to be set in your GitHub repository for the workflows to function correctly:

• AWS_ACCESS_KEY_ID: Your AWS Access Key ID.
• AWS_SECRET_ACCESS_KEY: Your AWS Secret Access Key.
• KUBECONFIG_SECRET: Your Kubernetes config file encoded in base64.

Before using the GitHub Actions workflows, you need to set up the AWS credentials as secrets in your GitHub repository. The included github_secrets.sh script automates the process of adding your AWS credentials to GitHub Secrets, which are then used by the workflows. To use this script:

1. Ensure you have the GitHub CLI (gh) installed and authenticated.

2. Run the script with the following command:

   ./github_secrets.sh

    

This script will:

• Extract your AWS Access Key ID and Secret Access Key from your local AWS configuration.
• Use the GitHub CLI to set these as secrets in your GitHub repository.

Note: It’s crucial to handle AWS credentials securely. The provided script is for demonstration purposes, and in a production environment, you should use a secure method to inject these credentials into your CI/CD pipeline.

These secrets are consumed by the GitHub Actions workflows to access your AWS resources and manage your Kubernetes cluster.

For the Continuous Deployment workflow to function properly, it requires access to your Kubernetes cluster. This access is granted through the KUBECONFIG file. You need to add this file manually to your GitHub repository’s secrets to ensure secure and proper deployment.

To add your KUBECONFIG to GitHub Secrets, follow these steps:

1. Encode your KUBECONFIG file to a base64 string:

   cat ~/.kube/config | base64

    

2. Copy the encoded output to your clipboard.

3. Navigate to your GitHub repository on the web.

4. Go to Settings > Secrets > New repository secret.

5. Name the secret KUBECONFIG_SECRET.

6. Paste the base64-encoded KUBECONFIG data into the secret’s value field.

7. Click Add secret to save the new secret.

This KUBECONFIG_SECRET is then used by the CD workflow to authenticate with your Kubernetes cluster and apply the required configurations.

Important: Be cautious with your KUBECONFIG data as it provides administrative access to your Kubernetes cluster. Only store it in secure locations, and never expose it in logs or to unauthorized users.

To access and manage the Database from your local machine, you can use k9s to port forward the service and then connect to it using BeeKeeper.

1. Open k9s in your terminal.
2. Navigate to the services section by typing :svc and pressing Enter.
3. Search for the service named postgres-service.
4. With the postgres-service highlighted, press Shift+F to set up port forwarding to your local machine.

Once you’ve port forwarded the postgres-service:

1. Open Beekeeper.

2. Connect to the Postgres Database using the localhost address and the port 5432.

3. Use the USERNAME you added in the k8s environment variable.

4. Get the PASSWORD from k8s secrets using k9s.

   • Navigate to secrets

   • Find db-password-secret

   • Tab on x in your keyboard to decode the generated password

Create a table in the database and insert data.

1. Run the following query to create table.

CREATE TABLE posts (
    id SERIAL PRIMARY KEY,
    title TEXT NOT NULL,
    body TEXT NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

2. Run the following query to insert data into the table.

INSERT INTO posts (title, body) VALUES ('First Post', 'This is the first post.');
INSERT INTO posts (title, body) VALUES ('Second Post', 'This is the second post.');
INSERT INTO posts (title, body) VALUES ('Third Post', 'This is the third post.');

4. Check the application webpage to see the updates.

After running the queries, you should be able to verify that the new entries have been added to the database and showed on the webpage.

In case you need to tear down the infrastructure and services that you have deployed, a script named destroy.sh is provided in the repository. This script will:

• Log in to Amazon ECR.
• Delete the specified Docker image from the ECR repository.
• Delete the Kubernetes deployment and associated resources.
• Delete the Kubernetes namespace.
• Destroy the AWS resources created by Terraform.

1. Open the destroy.sh script.

2. Ensure that the variables at the top of the script match your AWS and Kubernetes settings:

   REGION="REGION"
   REPOSITORY_NAME="ECR_REPOSITORY_NAME"

    

1. Save the script and make it executable:

   chmod +x destroy.sh

    

2. Run the script:

   ./destroy.sh

    

This script will execute another script ecr-img-delete.sh which will delete all the images on the ECR to make sure the ECR is empty then terraform commands to destroy all resources related to your deployment. It is essential to verify that the script has completed successfully to ensure that all resources have been cleaned up and no unexpected costs are incurred.