Project Setup Manual and Automated

➤ Java Bean with Examples
➤ How to Code a Game
➤ Array Programs in Java
➤ Java Inline Thread Creation
➤ Java Custom Exception
➤ Hibernate vs JDBC
➤ Object Relational Mapping
➤ Check Oracle DB Size
➤ Check Oracle DB Version
➤ Generation of Computers
➤ XML Pros & Cons
➤ Git Analytics & Its Uses
➤ Top Skills for Cloud Professional
➤ How to Hire Best Candidates
➤ Scrum Master Roles & Work
➤ CyberSecurity in Python
➤ Protect from Cyber-Attack
➤ Solve App Development Challenges
➤ Top Chrome Extensions for Twitch Users
➤ Mistakes That Can Ruin Your Test Metric Program

Project Setup Manual and Automated | We have a product called VProfile, a social networking site written in Java consisting of multiple services. We will deploy it on five virtual machines (VMs). Unlike WordPress, where Apache and MySQL run on the same VM, in VProfile, we will have five services running in five different VMs. These services are:

  • Nginx
  • Tomcat
  • RabbitMQ
  • Memcached
  • MySQL

In real-time projects, you should have a local copy of the product to perform experiments and testing.

Table of Contents

Tools Required

To set up and manage our virtual environment, we will use the following tools:

  • Oracle VM VirtualBox: For creating and managing virtual machines.
  • Vagrant: For automating the setup of the VMs.
  • Git Bash: For running shell commands.
  • VSCode/Any Text Editor: For editing configuration files and code.

The term stack refers to a collection of services working together to create a complete web application experience. We need to set up and configure these services on our VMs to ensure seamless integration and functionality.

Architecture Overview

Here’s a brief overview of our architecture:

  1. Nginx (Load Balancer):
  • The user opens a browser and enters the IP address of the load balancer (Nginx).
  • Nginx, a popular web server, will be configured to balance the load and route requests to the Tomcat server.
  1. Tomcat Server:
  • Nginx routes the user requests to the Tomcat server.
  • If the application needs external storage, NFS servers can be used for centralized storage.
  1. MySQL Database:
  • User login details and other data are stored in the MySQL database.
  • Tomcat communicates with MySQL to fetch and store user data.
  1. RabbitMQ (Message Broker):
  • RabbitMQ connects with Tomcat to handle messaging and queuing between different parts of the application.
  1. Memcached (Caching Server):
  • Before reaching the MySQL database, requests go through Memcached.
  • Memcached provides database caching to reduce load and improve performance.
  • User data is cached after the initial request to speed up subsequent requests.
VProfile+Project

When a user logs in, the application running on Tomcat sends an SQL query to access user information stored in MySQL. Before hitting the MySQL database, the request first checks Memcached for cached data. If the data is found in the cache, it’s returned quickly without querying MySQL. If not, the request goes to MySQL, and the data is cached for future requests.

By setting up these services across different VMs, we ensure efficient load balancing, effective data caching, and seamless communication between different components of the web application.

VM Setup

Clone the project. Switch to the local branch.

git clone https://github.com/hkhcoder/vprofile-project.git
cd .\vprofile-project\
git checkout local

First, we will do the manual setup of the vagrant and then we will automate it.

Prerequisite:-

  1. Oracle VM VirtualBox
  2. Git bash or equivalent editor
  3. Vagrant
  4. Vagrant plugins

Execute the below command in your computer to install hostmanager plugin:-

$ vagrant plugin install vagrant-hostmanager

In the vagrantFile you can find five VMS configurations. The first four VMs we use CentOS-9 but web01 which is nginx VM we will use Ubuntu.

Check whether any VMs are active, if active then go to that folder and destroy them:-

$ vagrant global-status --prune
$ cd  C:/install/vagrant-vms/finance
$ vagrant destroy --force
$ vagrant global-status --prune

id       name   provider state  directory
--------------------------------------------------------------------

Switch to the folder where the vagrant file is located, and start the VMS.

cd /c/workspace/vscode/devops/vproject/vprofile-project/
cd vagrant/Manual_provisioning_WinMacIntel/
vagrant up

Bringing up all the vm’s may take a long time based on various factors. If VM setup stops in the middle run “vagrant up” command again. You can also do vagrant reload or vagrant reload <vm-name>.

All the VM’s hostname and /etc/hosts file entries will be automatically updated.

$ vagrant global-status
id       name   provider   state   directory
--------------------------------------------------------------------------------------------------------------------------------
db93120  db01   virtualbox running C:/workspace/vscode/devops/vproject/vprofile-project/vagrant/Manual_provisioning_WinMacIntel
c211118  mc01   virtualbox running C:/workspace/vscode/devops/vproject/vprofile-project/vagrant/Manual_provisioning_WinMacIntel
e670938  rmq01  virtualbox running C:/workspace/vscode/devops/vproject/vprofile-project/vagrant/Manual_provisioning_WinMacIntel
a426db1  app01  virtualbox running C:/workspace/vscode/devops/vproject/vprofile-project/vagrant/Manual_provisioning_WinMacIntel
17f5832  web01  virtualbox running C:/workspace/vscode/devops/vproject/vprofile-project/vagrant/Manual_provisioning_WinMacIntel

Let us login to the web01, and see the /etc/hosts file:-

vagrant ssh web01
sudo -i
cat /etc/hosts

The /etc/hosts has the following entry:-

127.0.0.1       localhost

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost   ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
127.0.1.1       ubuntu-jammy    ubuntu-jammy

127.0.2.1 web01 web01

## vagrant-hostmanager-start
192.168.56.12   app01

192.168.56.15   db01

192.168.56.14   mc01

192.168.56.13   rmq01

192.168.56.11   web01

## vagrant-hostmanager-end

If you try to access a machine with the name app01, it is going to look for its IP. First, it is going to look for the IP in the host file. If it’s not found in the host file, then it will go and check the DNS of your ISP.

We did not add this entry, vagrant did it by using the vagrant-hostmanager plugin. So this is the plugin that we installed initially. In the vagrant file you can find the following entries that are responsible for creating these IP configurations:-

config.hostmanager.enabled = true 
config.hostmanager.manage_host = true

Just install the vagrant-hostmanager plugin, add these entries, and automatically your VMs will have the hostname for IP mapping. We can find these IP configurations in all the VMs.

ping app01 -c 4

You can log in to each VM, check for these IP configurations, and ping to the other services. If you see anything that does not give a response back, then you need to reboot that machine.

The Setup should be done in below mentioned order:-

  1. MySQL (Database SVC)
  2. Memcache (DB Caching SVC)
  3. RabbitMQ (Broker/Queue SVC)
  4. Tomcat (Application SVC)
  5. Nginx (Web SVC)

MySQL Setup

Login to the db vm

$ vagrant ssh db01
$ sudo -i

Verify Hosts entry, if entries missing update the it with IP and hostnames

# cat /etc/hosts

Update OS with latest patches

# dnf update -y

Set Repository

# dnf install epel-release -y

Install Maria DB Package

# dnf install git mariadb-server -y

Starting & enabling mariadb-server

# systemctl start mariadb
# systemctl enable MariaDB
# systemctl status mariadb

Hit q to quit. Notice that the package name is mariadb-server but the service name is mariadb.

RUN mysql secure installation script.

# mysql_secure_installation

Set db root password, We can use admin123 as password.

Set DB name and users.

# mysql -u root -padmin123
mysql> create database accounts;
mysql> grant all privileges on accounts.* TO 'admin'@'localhost' identified by 'admin123';
mysql> grant all privileges on accounts.* TO 'admin'@'%' identified by 'admin123';
mysql> FLUSH PRIVILEGES;
mysql> exit;

Download Source code & Initialize Database.

# cd /tmp/
# git clone -b local https://github.com/hkhcoder/vprofile-project.git
# cd vprofile-project
# ls src/main/resources/
# mysql -u root -padmin123 accounts < src/main/resources/db_backup.sql
# mysql -u root -padmin123 accounts
mysql> show tables;
+--------------------+
| Tables_in_accounts |
+--------------------+
| role               |
| user               |
| user_role          |
+--------------------+

mysql> exit;

Starting the firewall and allowing the mariadb to access from port no. 3306

# systemctl start firewalld
# systemctl enable firewalld
# firewall-cmd --get-active-zones
# firewall-cmd --zone=public --add-port=3306/tcp --permanent
# firewall-cmd --reload
# systemctl restart mariadb

Mamcache Setup

Login to the Memcache vm

$ vagrant ssh mc01
$ sudo -i

Verify Hosts entry, if entries missing update the it with IP and hostnames

# cat /etc/hosts

Update CentOS with latest patches (Optional), it will take time.

# dnf update -y

Install, start & enable memcache on port 11211

# sudo dnf install epel-release memcached -y
# cat /etc/sysconfig/memcached
# sed -i 's/127.0.0.1/0.0.0.0/g' /etc/sysconfig/memcached
# cat /etc/sysconfig/memcached
# sudo systemctl start memcached
# sudo systemctl enable memcached
# sudo systemctl status memcached

The command sed -i 's/127.0.0.1/0.0.0.0/g' /etc/sysconfig/memcached is used to modify the configuration file for Memcached. Here’s a breakdown:

  • sed: Stream Editor for filtering and transforming text.
  • -i: Edits the file in place, without creating a temporary file.
  • 's/127.0.0.1/0.0.0.0/g': The s command is used for substitution. It replaces all occurrences (g for global) of 127.0.0.1 with 0.0.0.0.
  • /etc/sysconfig/memcached: The file being edited.

Purpose: This command changes the binding address in the Memcached configuration from 127.0.0.1 (localhost) to 0.0.0.0 (all available IP addresses). This allows Memcached to accept connections from any IP address, not just localhost.

Starting the firewall (Optional)

# firewall-cmd --add-port=11211/tcp
# firewall-cmd --runtime-to-permanent
# firewall-cmd --add-port=11111/udp
# firewall-cmd --runtime-to-permanent

Allow the port 11211 to access Memcache:-

# sudo memcached -p 11211 -U 11111 -u memcached -d

This command is used to start a memcached server instance with specific options. Let’s break it down:

  • sudo: Runs the command with superuser (root) privileges.
  • memcached: The command to start the memcached server.
  • -p 11211: Specifies the TCP port on which memcached will listen for incoming connections (in this case, port 11211).
  • -U 11111: Specifies the UDP port on which memcached will listen for incoming connections (in this case, port 11111).
  • -u memcached: Runs the memcached process as the memcached user for security purposes.
  • -d: Runs memcached as a daemon, meaning it will run in the background.

Summary: The command starts a memcached server that:

  • Listens on TCP port 11211 for incoming connections.
  • Listens on UDP port 11111 for incoming connections.
  • Runs as the memcached user.
  • Runs in the background as a daemon process.

RabbitMQ Setup

Login to the RabbitMQ vm

$ vagrant ssh rmq01
$ sudo -i

Verify Hosts entry, if entries missing update it with IP and hostnames

# cat /etc/hosts

Update CentOS with the latest patches (Optional), it will take time.

# yum update -y

Set EPEL Repository

# yum install epel-release -y

Install Dependencies

# yum install wget -y
# cd /tmp/
# dnf -y install centos-release-rabbitmq-38
# dnf --enablerepo=centos-rabbitmq-38 -y install rabbitmq-server
# systemctl enable --now rabbitmq-server

The command systemctl enable --now rabbitmq-server is a combination of enabling and starting the RabbitMQ server. Here’s a breakdown:

  • systemctl enable rabbitmq-server: Configures the RabbitMQ service to start automatically at boot time.
  • systemctl start rabbitmq-server: Starts the RabbitMQ service immediately.

When you use the --now option with systemctl enable, it performs both actions in one step: systemctl enable --now rabbitmq-server: Enables the RabbitMQ service to start on boot and start it immediately. This command is convenient for ensuring that the service is both running now and will also start automatically in the future.

Setup access to user test and make it admin

# sh -c 'echo "[{rabbit, [{loopback_users, []}]}]." > /etc/rabbitmq/rabbitmq.config'
# rabbitmqctl add_user test test
# rabbitmqctl set_user_tags test administrator
# rabbitmqctl set_permissions -p / test ".*" ".*" ".*"
# systemctl restart rabbitmq-server
# systemctl status rabbitmq-server

Starting the firewall and allowing port 5672 to access RabbitMQ

# systemctl enable --now firewalld
# firewall-cmd --add-port=5672/tcp
# firewall-cmd --runtime-to-permanent
# firewall-cmd --reload
# systemctl restart rabbitmq-server

Tomcat Setup

Login to the Tomcat VM

$ vagrant ssh app01
$ sudo -i

Verify Hosts entry, if entries missing update the it with IP and hostnames

# cat /etc/hosts

Update CentOS with latest patches (Optional), it will take time.

# yum update -y

Part-1: Download and Install Tomcat

Set Repository

# yum install epel-release -y

Install Dependencies

# dnf install java-11-openjdk java-11-openjdk-devel git maven wget -y

Change dir to /tmp

# cd /tmp/

Download & Tomcat Package

# wget https://dlcdn.apache.org/tomcat/tomcat-10/v10.1.34/bin/apache-tomcat-10.1.34.tar.gz
# tar xzvf apache-tomcat-10.1.34.tar.gz
# ls apache-tomcat-10.1.34

Add tomcat user

# useradd --home-dir /usr/local/tomcat --shell /sbin/nologin tomcat

Copy data to tomcat home dir

# cp -r /tmp/apache-tomcat-10.1.34/* /usr/local/tomcat/

Make tomcat user owner of tomcat home dir

# chown -R tomcat.tomcat /usr/local/tomcat

Part-2: Setup systemctl command for tomcat

Create tomcat service file

# vi /etc/systemd/system/tomcat.service

Update the file with below content

[Unit]
Description=Tomcat
After=network.target

[Service]
User=tomcat
WorkingDirectory=/usr/local/tomcat
Environment=JRE_HOME=/usr/lib/jvm/jre
Environment=JAVA_HOME=/usr/lib/jvm/jre
Environment=CATALINA_HOME=/usr/local/tomcat
Environment=CATALINE_BASE=/usr/local/tomcat
ExecStart=/usr/local/tomcat/bin/catalina.sh run
ExecStop=/usr/local/tomcat/bin/shutdown.sh
SyslogIdentifier=tomcat-%i

[Install]
WantedBy=multi-user.target

Reload systemd files

# systemctl daemon-reload

Start & Enable service

# systemctl enable --now tomcat

Enabling the firewall and allowing port 8080 to access the tomcat

# systemctl enable --now firewalld
# firewall-cmd --get-active-zones
# firewall-cmd --zone=public --add-port=8080/tcp --permanent
# firewall-cmd --reload

Code Build and Deploy

In the app01 (where we have installed the tomcat), we will install Maven, build the code and deploy it on the server.

Download Source code

# cd
# git clone -b main https://github.com/hkhcoder/vprofile-project.git

Update configuration

# cd vprofile-project
# vim src/main/resources/application.properties

Update file with backend server details if you have changed anything from the default settings.

Build code: Run below command inside the repository (vprofile-project)

# ls
ansible  Jenkinsfile  pom.xml  README.md  src  vagrant

# mvn install

It will create a target folder, and inside that we can find the artifact vprofile-v2.war.

# ls
ansible  Jenkinsfile  pom.xml  README.md  src  target  vagrant

# ls target/
classes                 jacoco.exec     site              vprofile-v2
generated-sources       maven-archiver  surefire-reports  vprofile-v2.war
generated-test-sources  maven-status    test-classes

In the /usr/local/tomcat/webapps/ROOT folder we can find the default files of tomcat server, we need to replace them with vprofile-v2.war.

# ls /usr/local/tomcat/webapps/ROOT

Deploy the artifact

# systemctl stop tomcat
# rm -rf /usr/local/tomcat/webapps/ROOT

Make sure you are in the vprofile-project directory.

# cp target/vprofile-v2.war /usr/local/tomcat/webapps/ROOT.war
# chown tomcat.tomcat /usr/local/tomcat/webapps -R
# systemctl restart tomcat
# systemctl status tomcat

NGINX Setup

NGNI just acts like a load balancer. It takes the request and just forwards it to the Tomcat service.

Login to the Nginx vm

$ vagrant ssh web01
$ sudo -i

Verify Hosts entry, if entries missing update the it with IP and hostnames

# cat /etc/hosts

Update Ubuntu OS with latest patches (Optional):-

# apt update
# apt upgrade

Install nginx

# apt install nginx -y

Create Nginx conf file

# vi /etc/nginx/sites-available/vproapp

Update with below content

upstream vproapp {
    server app01:8080;
}

server {
    listen 80;

    location / {
        proxy_pass http://vproapp;
    }
}

Remove default nginx conf

# rm -rf /etc/nginx/sites-enabled/default

Create link to activate website

# ln -s /etc/nginx/sites-available/vproapp /etc/nginx/sites-enabled/vproapp

Restart Nginx

# systemctl restart nginx

Validate

Now all the setup is done, the IP address for web01 was 192.168.56.11 (you can verify it in Vagrantfile). Open the browser on host machine and open this IP address.

After verifying it, destroy all the VMs, we will automating all the configurations. Make sure you are in the directory where Vagrantfile is located:-

$ vagrant global-status
$ vagrant destroy --force
$ vagrant global-status

Automating the Setup of the Entire VProfile Stack

We will walk you through setting up the entire VProfile stack automatically with just a single command.

By running the vagrant up command, all virtual machines (VMs) will be provisioned automatically. This includes setting up the following services:

  • Nginx
  • Tomcat
  • RabbitMQ
  • Memcached
  • MySQL

Bash Scripts for Automated Provisioning:- Our setup design is similar to the previous project, but with an added layer of automation. Bash scripts will be executed automatically to provision all these services. This means that the entire stack will be ready to go without any manual intervention.

Steps to Follow:

  1. Run vagrant up: This command will initiate the provisioning process.
  2. Automatic VM Provisioning: All VMs will be created and configured.
  3. Service Provisioning: The bash scripts will automatically set up Nginx, Tomcat, RabbitMQ, Memcached, and MySQL.

In the vprofile-project-main\vprofile-project-main\vagrant\Automated_provisioning_WinMacIntel folder, you can find the Vagrantfile and scripts related to each services/VMs. In each VMs you can find the path of the script.

db01.vm.provision "shell", path: "mysql.sh"

Open Git Bash, go the vprofile-project-main\vprofile-project-main\vagrant\Automated_provisioning_WinMacIntel directory, and initiate the provisioning process.

  • Tomcat is an application server to host Java Web Application like vprofile.
  • Nginx is a frontent server, web server and can be used as a Load balancer.
  • Mysql is a SQL Database server, similar are Mariadb, MSSql etc
  • RabbitMQ is the most widely deployed open source message broker.
  • Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls.

If you enjoyed this post, share it with your friends. Do you want to share more information about the topic discussed above or do you find anything incorrect? Let us know in the comments. Thank you!

Leave a Comment

Your email address will not be published. Required fields are marked *