How To Install Graylog Server using Puppet Automation

How To Install Graylog Server using Puppet Automation

This guide takes a deep illustration of how to install the Graylog Server on Ubuntu / Debian / CentOS using Puppet. Graylog is a free and open-source tool used to manage and aggregate logs. It is used to store, analyze and send alerts based on the logs collected. This tool is normally used to analyze both structured and unstructured data.

The Graylog server comprises the following components:

• MongoDB – stores the data and configurations.
• Graylog Server– The server that passes logs for visualization using the built-in web interface.
• ElasticSearch– this is the log analysis tool for the Graylog Server.
• Java /OpenJDK– offers the runtime environment for ElasticSearch.

Puppet is used here to automate the installation of all these components on desired nodes. This product, developed by Puppet Labs can be used to configure, manage and deploy the Graylog server.

#1 Install and Configure Puppet on Ubuntu / Debian / CentOS

This guide requires one to have Puppet installed and configured on their system. This involves setting up the Puppet server and the agent nodes attached to it. The latest version of Graylog is compatible with Puppet >= 6.21.0 < 8.0.0 which can be installed using the dedicated guides below:

Once you’ve configured Puppet server and client, validate connectivity

$ sudo /opt/puppetlabs/bin/puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for rocky-linux-8.localdomain
Info: Applying configuration version '1651833223'
Info: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml
Notice: Applied catalog in 0.01 seconds

#2 – Install Required Puppet Modules

For this guide, we will not only install the Graylog module but also other modules for dependencies like Java, MongoDB, and Elasticsearch.

Install Graylog Puppet module

We will begin by installing the Graylog module

$ sudo /opt/puppetlabs/bin/puppet module install graylog/graylog
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppet.com ...
Notice: Installing -- do not interrupt ...
/etc/puppetlabs/code/environments/production/modules
└─┬ graylog-graylog (v1.0.0)
  ├── puppetlabs-apt (v8.3.0)
  └── puppetlabs-stdlib (v7.1.0)

The Graylog module comes along with other required modules:

• The Puppet APT module – can as well be installed using the command:

sudo /opt/puppetlabs/bin/puppet module install puppetlabs-apt --version <version-number>

• The Puppet standard library module – can as well be installed as below:

sudo /opt/puppetlabs/bin/puppet module install puppetlabs-stdlib --version <version-number>

Install MongoDB Puppet module

Next, install the MongoDB module.

$ sudo /opt/puppetlabs/bin/puppet module install puppet-mongodb
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppet.com ...
Notice: Installing -- do not interrupt ...
/etc/puppetlabs/code/environments/production/modules
└─┬ puppet-mongodb (v4.1.1)
  ├─┬ puppet-systemd (v3.8.0)
  │ └── puppetlabs-inifile (v5.2.0)
  ├─┬ puppet-zypprepo (v4.0.1)
  │ └── puppetlabs-concat (v7.1.1)
  ├── puppetlabs-apt (v8.3.0)
  └── puppetlabs-stdlib (v7.1.0)

Unfortunately, this module is not fully compatible with Debian 11 systems. So in case you have Debian 11 nodes, you might face errors during the installation.

The other module to install is the Java module. It can be installed with the command:

$ sudo /opt/puppetlabs/bin/puppet module install puppetlabs-java
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules .
Notice: Downloading from https://forgeapi.puppet.com ...
Notice: Installing -- do not interrupt ...
/etc/puppetlabs/code/environments/production/modules
└─┬ puppetlabs-java (v8.0.0)
  ├── puppet-archive (v6.0.2)
  └── puppetlabs-stdlib (v7.1.0)

Finally, install the Elasticsearch module.

$ sudo /opt/puppetlabs/bin/puppet module install puppet-elasticsearch
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppet.com ...
Notice: Installing -- do not interrupt ...
/etc/puppetlabs/code/environments/production/modules
└─┬ puppet-elasticsearch (v8.0.2)
  ├─┬ puppet-elastic_stack (v8.0.0)
  │ ├─┬ puppet-yum (v5.4.0)
  │ │ └── puppetlabs-concat (v7.1.1)
  │ ├── puppetlabs-apt (v8.3.0)
  │ └── puppetlabs-stdlib (v7.1.0)
  ├── puppetlabs-java (v8.0.0)
  └── richardc-datacat (v0.6.2)

List the installed Puppet modules with the command:

$ sudo /opt/puppetlabs/bin/puppet module list --environment production
/etc/puppetlabs/code/environments/production/modules
├── graylog-graylog (v1.0.0)
├── puppet-archive (v6.0.2)
├── puppet-elastic_stack (v8.0.0)
├── puppet-elasticsearch (v8.0.2)
├── puppet-mongodb (v4.1.1)
├── puppet-systemd (v3.8.0)
├── puppet-yum (v5.4.0)
├── puppet-zypprepo (v4.0.1)
├── puppetlabs-apt (v8.3.0)
├── puppetlabs-concat (v7.1.1)
├── puppetlabs-inifile (v5.2.0)
├── puppetlabs-java (v8.0.0)
├── puppetlabs-stdlib (v7.1.0)
└── richardc-datacat (v0.6.2)
/etc/puppetlabs/code/modules (no modules installed)
/opt/puppetlabs/puppet/modules (no modules installed)

#3 – Install Graylog Server on Ubuntu / Debian / CentOS using Puppet

Once the modules have been installed, we will proceed and create a manifest to manage MongoDB, Elasticsearch, and Graylog on a single node as below.

Getting Package versions:

• Graylog releases
• System requirements (Supported version of Elasticsearch, MongoDB, Java)

Create file as below.

sudo vim /etc/puppetlabs/code/environments/production/manifests/init.pp

The file will contain the below lines:

class { 'mongodb::globals':
  manage_package_repo => true,
}->
class { 'mongodb::server':
  bind_ip => ['127.0.0.1'],
  ensure     => 'present',
  restart    => true,
}

include ::java

class { 'elasticsearch':
  ensure => 'present',
  status => 'enabled',
  version      => '7.10.2',
  restart_on_change => true,
  config => {
    'cluster.name' => 'graylog',
    'network.host' => '127.0.0.1',
  },
  jvm_options => [
    '-Xms512m',
    '-Xmx512m'
  ]
}

class { 'graylog::repository':
  version => '4.2'
}->
class { 'graylog::server':
  package_version => 'latest',
  config          => {
    'password_secret' => 'pmHuefc3sMv6SWN6wPoCss6hTy8vksYr1QkFmtVjChi1rdRr6s7FeqNJOrWOWlipMsfmFgqGJM8HLdpF3thwFA4QvLSPhC0O', # Fill in your password secret
    'root_password_sha2' => '434e27fac24a15cbf8b160b7b28c143a67d9e6939cbb388874e066e16cb32d75',# Fill in your root password hash
    'http_bind_address' => '0.0.0.0:9000',
     'http_external_uri'   => 'https://0.0.0.0:9000/',
  }
}

In the file, replace the values for password_secret generated with pwgenas below:

$ pwgen -N 1 -s 96
pmHuefc3sMv6SWN6wPoCss6hTy8vksYr1QkFmtVjChi1rdRr6s7FeqNJOrWOWlipMsfmFgqGJM8HLdpF3thwFA4QvLSPhC0O

root_password_sha2 is the generated the sha256 password for the administrator user:

$ echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1
Enter Password: Str0ngPassw0rd

Sample output:

434e27fac24a15cbf8b160b7b28c143a67d9e6939cbb388874e066e16cb32d75

On Debian/Ubuntu systems, you might get an error with MongoDB when applying the manifest. This error can be solved by editing the manifest to use a manually added MongoDB repository.

For example, MongoDB 4.4 repository can be added to Debian Buster with the commands:

wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
echo "deb http://repo.mongodb.org/apt/debian buster/mongodb-org/4.4 main"|sudo tee /etc/apt/sources.list.d/mongodb-org.list
sudo apt update

Once the repository and its GPG keys are added and the system updated, edit the manifest as below.

 class { 'mongodb::globals':
  manage_package_repo => false,
  manage_package      => true,
}->
class { 'mongodb::server':
  bind_ip => ['127.0.0.1'],
  ensure     => 'present',
  restart    => true,
}->
class { 'mongodb::client':
}
....

Finally, run the manifest on the agent as below:

sudo /opt/puppetlabs/bin/puppet agent -t

Sample Output:

Verify if Elasticsearch is running on port 9200:

$ curl -X GET localhost:9200
{
  "name" : "graylog.example.com",
  "cluster_name" : "graylog",
  "cluster_uuid" : "tMJGsHuNS6OUgCk5q8RGBQ",
  "version" : {
    "number" : "7.9.3",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "c4138e51121ef06a6404866cddc601906fe5c868",
    "build_date" : "2020-10-16T10:36:16.141335Z",
    "build_snapshot" : false,
    "lucene_version" : "8.6.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

#4 – Access Graylog Web interface

AT this point, the Graylog server is listening on port 9000. Allow this port through the firewall:

##For Firewalld
sudo firewall-cmd --add-port=9000/tcp --permanent
sudo firewall-cmd --reload

##For UFW
sudo ufw allow 9000/tcp

Now access the Graylog web interface using the URL http://IP_Address:9000

Provide the login credentials as above, the default username is admin and the password is the one set with the root_password_sha2. For this case, the password is Str0ngPassw0rd.

On successful authentication, you will be able to see the below Graylog dashboard.

That is it! Proceed and create dashboards to visualize the collected logs on the Graylog web interface.

Verdict

That marks the end of this guide on how to install Graylog Server on Ubuntu / Debian / CentOS using Puppet. We can now agree that Puppet automation makes it easy to install and configure the Graylog Server. I hope this was significant.

See more: