Stage 1:

1. Create Instances:

   1. Edit security group:

2. Create 3 VMs: Master, Slave-1, Slave-2

• To create an Ubuntu EC2 instance in AWS, follow these steps:

1. Sign in to the AWS Management Console:

   • Go to the AWS Management Console at https://aws.amazon.com/console/.

   • Sign in with your AWS account credentials.

2. Navigate to EC2:

   • Once logged in, navigate to the EC2 dashboard by typing "EC2" in the search bar at the top or by selecting "Services" and then "EC2" under the "Compute" section.

3. Launch Instance:

   • Click on the "Instances" link in the EC2 dashboard sidebar.

   • Click the "Launch Instance" button.

4. Choose an Amazon Machine Image (AMI):

   • In the "Step 1: Choose an Amazon Machine Image (AMI)" section, select "Ubuntu" from the list of available AMIs.

   • Choose the Ubuntu version you want to use. For example, "Ubuntu Server 20.04 LTS".

   • Click "Select".

5. Choose an Instance Type:

   • In the "Step 2: Choose an Instance Type" section, select the instance type that fits your requirements. The default option (usually a t2.micro instance) is suitable for testing and small workloads.

   • Click "Next: Configure Instance Details".

6. Configure Instance Details:

   • Optionally, configure instance details such as network settings, subnets, IAM role, etc. You can leave these settings as default for now.

   • Click "Next: Add Storage".

7. Add Storage:

   • Specify the size of the root volume:25 (default is usually fine for testing purposes).

   • Click "Next: Add Tags".

8. Add Tags:

   • Optionally, add tags to your instance for better organization and management.

   • Click "Next: Configure Security Group".

9. Configure Security Group:

   • In the "Step 6: Configure Security Group" section, configure the security group to allow SSH access (port 22) from your IP address.

   • You may also want to allow other ports based on your requirements (e.g., HTTP, HTTPS) as in this pic

   • Click "Review and Launch".

10. Review and Launch:

    • Review the configuration of your instance.

    • Click "Launch".

11. Select Key Pair:

    • In the pop-up window, select an existing key pair or create a new one.

    • Check the acknowledgment box.

    • Click "Launch Instances".

2. Access your instance:

   • Download tool- MobaXterm Xserver with SSH, telnet, RDP, VNC and X11 - Home Edition

   • install it.

   • settings → configuration→ ssh → checked ssh keepalive

   • click on session → Give ip address of master → username: Ubuntu

     session name: Master

   • Create duplicate session for slave-1, slave-2, Dont forget to enter their ip address.

   • open master, slave-1, slave-2

     

   • Enter this command on 3 instance.

     #sudo su

3. Setup k8s cluster":

   • Create 1.sh on all three instance.

     #vi 1.sh

     Enter below data:

       #Update System Packages [On Master & Worker Node]
       sudo apt-get update
     
       #Install Docker[On Master & Worker Node]
       sudo apt install docker.io -y
       sudo chmod 666 /var/run/docker.sock
     
       #Install Required Dependencies for Kubernetes[On Master & Worker Node]
       sudo apt-get install -y apt-transport-https ca-certificates curl gnupg
       sudo mkdir -p -m 755 /etc/apt/keyrings
     
       #Add Kubernetes Repository and GPG Key[On Master & Worker Node]
       curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
       echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
     
       #Give permission to execute:
       chmod +x 1.sh
       ls
     
       #run the file
       ./1.sh
     
       #Update Package List[On Master & Worker Node]
       sudo apt update
     
       #Install Kubernetes Components[On Master & Worker Node]
       sudo apt install -y kubeadm=1.28.1-1.1 kubelet=1.28.1-1.1 kubectl=1.28.1-1.1
     

       #Initialize Kubernetes Master Node [On MasterNode]
       sudo kubeadm init --pod-network-cidr=10.244.0.0/16
     

   • you will get output like this:

     

   • and copy this and run on worker nodes i.e. slave-1, slave-2. You can save this command if you want to add another virtual machine as a worker node you can use this.

   • You will get output like this on both slave

     

       #Configure Kubernetes Cluster [On MasterNode]
       mkdir -p $HOME/.kube
     
       #containing all info about k8s
       sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
       sudo chown $(id -u):$(id -g) $HOME/.kube/config
     
       #Deploy Networking Solution (Calico) [On MasterNode]
       kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
     
       #Deploy Ingress Controller (NGINX) [On MasterNode]
       kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.49.0/deploy/static/provider/baremetal/deploy.yaml
     

       #to see all worker node
       kubectl get nodes
     

     

4. Security scan of k8s cluster:

   • Visit this site: github.com/shopify/kubeaudit/releases

   • copy this link https://github.com/Shopify/kubeaudit/releases/download/v0.22.2/kubeaudit_0.22.2_linux_amd64.tar.gz

       #download this on master node
       wget https://github.com/Shopify/kubeaudit/releases/download/v0.22.2/kubeaudit_0.22.2_linux_amd64.tar.gz
     
       tar -xvzf kubeaudit_0.22.2_linux_amd64.tar.gz
     
       #move this executable file:
       sudo mv kubeaudit /usr/local/bin/
     
       #to audit Kubernetes clusters against common security
       kubeaudit all
     

5. Create VM for jenkins, sonar and nexus.

   Create 2 instance for sonarcube server and nexus server.

   AMI: ubuntu server 20.04LTS

   Instance type: t2.medium

   key: key.pem

   security group: default

   storage: 20

   launch instance

   name: SonarQube, Nexus

   Create instance for jenkins:

   AMI: ubuntu server 20.04LTS

   Instance type: t2.large

   key: key.pem

   security group: default

   storage: 30

   launch instance

   name: jenkins

   launch instance

   Create duplicate session for three of them in MobaXterm:

   Enter ip address, session name.

6. Setup sonarQube and Nexus:

    # Update package manager repositories
    sudo apt-get update
   
    # Install necessary dependencies
    sudo apt-get install -y ca-certificates curl
   
    # Create directory for Docker GPG key
    sudo install -m 0755 -d /etc/apt/keyrings
   
    # Download Docker's GPG key
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
   
    # Ensure proper permissions for the key
    sudo chmod a+r /etc/apt/keyrings/docker.asc
   
    # Add Docker repository to Apt sources
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
    $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
   
    # Update package manager repositories
    sudo apt-get update
   
    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin 
   
    sudo chmod 666 /var/run/docker.sock
   
    #check if docker is present or not.
    docker pull hello-world
   
    #Create Sonarqube Docker container
    #To run SonarQube in a Docker container with the provided command, you can follow these steps:
    #Run the following command on SonarQube instance:
    docker run -d --name sonar -p 9000:9000 sonarqube:lts-community
    docker ps
   
    #Create Nexus using docker container
    #Run the following command on Nexus instance:
    docker run -d --name Nexus -p 8081:8081 sonartype/nexus3
    docker ps
   

   Run SonarQube:

   take sonarQube instance ip:port_number and run it on browser

   

   • default username and password is: admin. You can update later.

Run Nexus:

take Nexus instance ip:port_number and run it on browser

• to login username is admin and password is located in /nexus-data/admin.password

• Go to nexus server:

    docker ps
    docker exec -it <containerid> /bin/bash
  

  

  • copy this selected part and that will be your password. You can update the password later.

  • check the box enable anonymous access.

7. Setup jenkins:

    #Installing Jenkins on Ubuntu
    sudo apt update
   
    # Install OpenJDK 17 JRE Headless
    sudo apt install openjdk-17-jre-headless -y
   
    # Download Jenkins GPG key
    sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
      https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
   
    # Add Jenkins repository to package manager sources
    echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
      https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
      /etc/apt/sources.list.d/jenkins.list > /dev/null
   
    # Update package manager repositories
    sudo apt-get update
   
    # Install Jenkins
    sudo apt-get install jenkins -y
   
    # Update package manager repositories
    sudo apt-get update
   
    # Install necessary dependencies
    sudo apt-get install -y ca-certificates curl
   
    # Create directory for Docker GPG key
    sudo install -m 0755 -d /etc/apt/keyrings
   
    # Download Docker's GPG key
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
   
    # Ensure proper permissions for the key
    sudo chmod a+r /etc/apt/keyrings/docker.asc
   
    # Add Docker repository to Apt sources
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
    $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
   
    # Update package manager repositories
    sudo apt-get update
   
    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
   

   Run jenkins:

   Take Nexus instance ip:port_number and run it on browser.

   

   • to get a password copy /var/lib/jenkins/secrets/initialAdminPassword and paste in shell.

• Customize jenkins → install suggested plugins → create user → save password for later use. → jenkins is ready.

Stage 2:

1. Create a Private Git Repository:

   • Go to your preferred Git hosting platform (e.g., GitHub, GitLab, Bitbucket).

   • Log in to your account or sign up if you don't have one.

   • Create a new repository and set it as private.

2. Generate a Personal Access Token:

   • Navigate to your account settings or profile settings.

   • Look for the "Developer settings" or "Personal access tokens" section.

   • Generate a new token, providing it with the necessary permissions (e.g., repo access).

3. Clone the Repository Locally:

   • Open Git Bash or your terminal.

   • Navigate to the directory where you want to clone the repository.

   • Use the git clone command followed by the repository's URL. For example:

       git clone <repository_URL>
     

Replace <repository_URL> with the URL of your private repository.

4. Add Your Source Code Files:

   • Navigate into the cloned repository directory.

   • Paste your source code files or create new ones inside this directory.

5. Stage and Commit Changes:

   • Use the git add command to stage the changes:

       git add .
     

   • Use the git commit command to commit the staged changes along with a meaningful message:

       git commit -m "done"
     

6. Push Changes to the Repository:

   • Use the git push command to push your committed changes to the remote repository:

       git push
     

   • If it's your first time pushing to this repository, you might need to specify the remote and branch:

       git push -u origin master
     

Replace master with the branch name if you're pushing to a different branch.

7. Enter Personal Access Token as Authentication:

   • When prompted for credentials during the push, enter your username (usually your email) and use your personal access token as the password

Stage 3:

1. Install Plugins for CI-CD pipeline

• Go to jenkins. Run it on browser.

• Manage jenkins → plugin → available plugin

  

  • Plugins: Eclipse Temurin installer, config file provider, maven integration, SonarQube Scanner, docker, docker pipeline, docker-build-step, kubernetes, kubernetes CLI, kubernetes client API, Kubernetes credentials

  • Install it.

2. Configure Plugins:

   • Configure Plugin: manage plugin → tools →

     1. add jdk → name: jdk17 → select install automatically → add installer → install from adoptium.net

        

     2. add SonarQube Scanner → name: sonar-scanner → install auto.

        

     3. add maven→ name: maven3 → install auto.

        

     4. add docker→ name: docker → install auto.→ select download from docker.com → apply

        

3. Create pipeline

   • New item → name: Boardgame → select pipeline → ok → configure → discard old builds → max of build to keep: 2

• Write script: select hello world template.

• Add creddentials for github, sonarQube

  For github: go to manage jenkins→ credential → system → global credential → add credential → username:neha-sawant7 → password: token → id:giev any → create

  For SonarQube: go to manage jenkins→ credential → system → global credential → add credential → kind: secret text→ go to sonarqube server → Administration → security → users → tokens → name: sonar-token → generate → copy→ paste in credential section in jenkins→ id: sonar-token → create

Pipeline script:

Stage: Git checkout

• If you dont know how to write script. select pipeline syntax.

  

• To generate script for git checkout:

  Steps→ sample step: select git → repo url:give your repo url → branch:main → credential: add → go to pipeline syntax → sample step: git → repo:https://github.com/neha-sawant7/Board-game.git → branch:main → select added creditials → generate pipeline script

  

  

  Stage: compile

  

  Stage: test

  

  Stage: File System Scan

• To install trivy:

• run below commands on jenkins instance:

  

    trivy --version
  

  

  stage: SonarQube Analsyis

• Configure sonarQube server:

  Manage jenkins → system → SonarQube server → add → name:sonar → url:ipaddress of sonarqube server(dont froget to remove last slash) → server auth token: sonar-token → save

• To generate script for sonarqube:

  pipeline syntax → sample step: sonarqube → server auth token: sonar-token→ generate pipeline script

  

  

  Stage: Quality Gate

• To write quality gate:

  go to sonarcube site→ administration → configuration → webhook → create → name: jenkins→ URL: as in below snip

  

  Stage: Build

  

  Stage: Publish To Nexus

• manage jenkins → managed files → add new config → select global maven setting.xml → id:global settings → next → you will get content → scroll until you find server section →

  

  • To generate script for Maven :

    pipeline syntax → sample step: withmaven → maven: maven3 JDK: jdk17 → global maven setting config: myglobalsetting → generate pipeline script

    Stage: Build & Tag Docker Image

  • To generate script for docker :

    pipeline syntax → sample step: withdockerregistry → add cred: add → username → password which we setup for jenkins alreadyID: docker-cred → select ad registry cred → docker installation: docker → generate pipeline script

  • save pipeline.

    

    Stage: Deploy To Kubernetes

  • Create service account:

  • go to master instance→ vi svc.yaml → content down here.

            apiVersion: v1
            kind: ServiceAccount
            metadata:
              name: jenkins
              namespace: webapps

• Create namespace:

    kubectl create ns webapps
    kubectl apply -f svc.yaml
  

• Create role: vi role.yaml

    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: app-role
      namespace: webapps
    rules:
      - apiGroups:
            - ""
            - apps
            - autoscaling
            - batch
            - extensions
            - policy
            - rbac.authorization.k8s.io
        resources:
          - pods
          - secrets
          - componentstatuses
          - configmaps
          - daemonsets
          - deployments
          - events
          - endpoints
          - horizontalpodautoscalers
          - ingress
          - jobs
          - limitranges
          - namespaces
          - nodes
          - pods
          - persistentvolumes
          - persistentvolumeclaims
          - resourcequotas
          - replicasets
          - replicationcontrollers
          - serviceaccounts
          - services
        verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  

    kubectl apply -f role.yaml
  

• Bind the role to service account: vi bind.yaml

    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: app-rolebinding
      namespace: webapps 
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: app-role 
    subjects:
    - namespace: webapps 
      kind: ServiceAccount
      name: jenkins
  

    kubectl apply -f bind.yaml
  

• Now we want to connect jenkins to k8s cluster. So need to create token for auth.

• vi sec.yaml

    apiVersion: v1
    kind: Secret
    type: kubernetes.io/service-account-token
    metadata:
      name: mysecretname
      annotations:
        kubernetes.io/service-account.name: jenkins
  

    kubectl apply -f sec.yaml -n webapps
    kubectl describe secret mysecretname -n webapps
  

  

  • we have got token

• Add credentials for jenkins to k8s

  go to manage jenkins→ credential → system → global credential → add credential → kind: secret text → secret: is mention in above snip →ID: k8-cred→

  description: k8-cred→ create

• To generate script for k8 :

  configure →pipeline syntax → sample step: withkubeconfig → cred : k8-cred → kubernetes server endpoint:https://172.31.8.146:6443 → cluster:kubernetes → namespace: webapps → generate pipeline script

• to get kubernetes server endpoint go to master node.

  #cd ~/.kube

  #ls

  #cat config

  Server endpoint and cluster name as below in snip.

  

  

• Write deployment-service.yaml

  ```bash apiVersion: apps/v1 kind: Deployment # Kubernetes resource kind we are creating metadata: name: boardgame-deployment spec: selector: matchLabels: app: boardgame replicas: 2 # Number of replicas that will be created for this deployment template: metadata: labels: app: boardgame spec: containers:

  • name: boardgame image: adijaiswal/boardshack:latest # Image that will be used to containers in the cluster imagePullPolicy: Always ports:
    • containerPort: 8080 # The port that the container is running on in the cluster

apiVersion: v1 # Kubernetes API version kind: Service # Kubernetes resource kind we are creating metadata: # Metadata of the resource kind we are creating name: boardgame-ssvc spec: selector: app: boardgame ports:

• protocol: "TCP" port: 80 targetPort: 8080 type: LoadBalancer # type of the service. ```

• install kubectl on jenkins:

  go to jenkins instance → enter below commands:

    curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.19.6/2021-01-05/bin/linux/amd64/kubectl
    chmod +x ./kubectl
    sudo mv ./kubectl /usr/local/bin
    kubectl version --short --client
  

  

  

  4. Enable mail notification:

• go to google account → security → 2 step verification → enter password → app passwords → app name:jenkins create → generated app password copy password

• manage jenkins → system → scroll down → extended email notification→ SMTP server: smtp.gmail.com → SMTP port: 465 → cred → add → tick use SSL → add →jenkins → username: emailid → password: mention in above image → ID: mail-cred → description: mail-cred→ add → now select added cred →

• scroll down → email notification→ SMTP server:smtp.gmail.com →advanced → tick use SSl → SMTP port:465 → tick SMTP auth → username:emailid → password: as mention in above snip

• tick test configuration by sending test email→ test e-mail recipient:emailid →click test onfiguration → check email box

  

• boardgame→ configuration → pipeline →write script for mail notification

    post {
        always {
            script {
                def jobName = env.JOB_NAME
                def buildNumber = env.BUILD_NUMBER
                def pipelineStatus = currentBuild.result ?: 'UNKNOWN'
                def bannerColor = pipelineStatus.toUpperCase() == 'SUCCESS' ? 'green' : 'red'
  
                def body = """
                    <html>
                    <body>
                    <div style="border: 4px solid ${bannerColor}; padding: 10px;">
                    <h2>${jobName} - Build ${buildNumber}</h2>
                    <div style="background-color: ${bannerColor}; padding: 10px;">
                    <h3 style="color: white;">Pipeline Status: ${pipelineStatus.toUpperCase()}</h3>
                    </div>
                    <p>Check the <a href="${BUILD_URL}">console output</a>.</p>
                    </div>
                    </body>
                    </html>
                """
  
                emailext (
                    subject: "${jobName} - Build ${buildNumber} - ${pipelineStatus.toUpperCase()}",
                    body: body,
                    to: 'nehasawant719@gmail.com',
                    from: 'jenkins@example.com',
                    replyTo: 'jenkins@example.com',
                    mimeType: 'text/html',
                    attachmentsPattern: 'trivy-image-report.html'
                )
            }
        }
    }
  

  5. Run this pipeline

we have artifact and trivy report also. You can check sonarQube also.-

• Run on browser:

  

  slave-1 ip:port

  

  

  

  

Phase 4:

1. create server for monitoring.

   name: monitor

   AMI: ubuntu 22.04

   Instance type: t2.large

   key: key.pem

   security group: default

   storage:20

   Launch instance.

2. Connect vm on Mobxterm.

    sudo apt update
   

3. Install prometheus

    wget  https://github.com/prometheus/prometheus/releases/download/v3.5.0/prometheus-3.5.0.linux-amd64.tar.gz
    ls
    #you will get prometheus version
    tar -xvf promethues-3.5-rc.0.linux-amd64.tar.gz
    rm -rd promethues-3.5-rc.0.linux-amd64.tar.gz
    ls
    you will get prometheus
    cd promethues-3.5-rc.0.linux-amd64/
    ls
    ./prometheus &
   

4. Access prometheus on browser:

   

5. Install Grafana:

   visit: Download Grafana | Grafana Labs

   select linux:

    sudo apt-get install -y adduser libfontconfig1 musl
    wget https://dl.grafana.com/enterprise/release/grafana-enterprise_12.1.0_amd64.deb
    sudo dpkg -i grafana-enterprise_12.1.0_amd64.de
   

   

    sudo /bin/systemctl start grafana-serve
   

6. run it on browser:

   

   5. Default username and password is: admin

   6. Install blackbox exporter:

       wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.27.0/blackbox_exporter-0.27.0.linux-amd64.tar.gz
       tar -xvf blackbox_exporter-0.27.0.linux0amd64.tar.gz
       ls
       #yo u will see tar file remove that
       rm -rf blackbox_exporter-0.27.0.linux0amd64.tar.gz
       ls
       cd blackbox_exporter-0.27.0.linux0amd64/
       ./blackbox_exporter &
      

8. Edit prometheus.yaml

   

    scrape_configs:
      - job_name: 'blackbox'
        metrics_path: /probe
        params:
          module: [http_2xx]  # Look for a HTTP 200 response.
        static_configs:
          - targets:
            - http://prometheus.io    # Target to probe with http.
            - http://13.234.66.128:30119 9#where our application is running
        relabel_configs:
          - source_labels: [__address__]
            target_label: __param_target
          - source_labels: [__param_target]
            target_label: instance
          - target_label: __address__
            replacement: 127.0.0.1:9115  # The blackbox exporter's real hostname:port.
   

9. Restart prometheus:

    pgrep prometheus
    #you will get id of process running, kill that id
    kill <id>
   
    #restart prometheus
    ./prometheus &
   

Now our app up and running.

10. Add data source as prometheus in grafana.

    Grafana→ connections → data source → add data source → promethues → provide ip address of prometheus. i.e. 12.232.255.214:9000

    

    

    save test.

11. Import dashboard

    

    Go to Prometheus Blackbox Exporter | Grafana Labs.

    Copy the dashboard

    

    Paste here and and select the data source : prometheus then click on import

    

    You will see the all charts here.

    

    Here you can select targets you want to monitor:

    

12. System level monitoring:

    jenkins → manage jenkins → plugins → search prometheus → install it. → if doesnt work restart server.

    go to jenkins instance → install node exporter

    wget https://github.com/prometheus/node_exporter/releases/download/v1.9.1/node_exporter-1.9.1.linux-amd64.tar.gz
    

    

    

    Run it on browser:

    

    Now go to monitor instance.

    

    edit the yaml file:

    

    pgrep prometheus
    kill <id>
    ./prometheus &
    

    

    Take dashboard from browser and add id load it and give data source: prometheus.