Graylog provides a simple and (relative) small environment, that can analyze different type of logs. Graylog offers some commercial plans but in general it is open source and can be used by everyone for free. This documentation is for Graylog Open version 6.0 (further called just “Graylog”) following the official documentation with some adjustments. It is for internal use or labs only. See chapter Hardening for further hardening of the systems.

Requirements

Graylog is NOT a small simple tool that can just be downloaded and executed. It is a set of minimum two servers where the recommended lowest system requirements (for 1-10GB logs/day) are: Server 1 (Graylog, MongoDB):

CPU: 8
RAM: 16GB
Storage: 130GB

Server 2 (OpenSearch):

CPU: 8
RAM: 24GB
Storage: Depends on the log amount. Start with 100GB but be ready to increase.

In general SSDs with minimum of 50.000 IOPS is required but 300.000 IOPS is recommended for OpenSearch. More details can be found at the Graylog website: Planning Your Deployment (graylog.org)

BUT: I want to use this in a small setup and don’t have the recommended requirements, so I will not fulfill them. I will use 4 CPUs and 8 GB of memory and use just one server!

Additionally to the server resources, you need a already running Elasticsearch node or cluster Graylog can connect to.

Basic Operating System setup

I chose Debian 12 to run the MongoDB and OpenSearch Server. Install Debian 12 and all available updates and sudo for security reasons. Set the timezone by running this command (it is recommend to use UTC):

sudo timedatectl set-timezone UTC

Install MongoDB

Follow these steps to install MongoDB with required dependencies:

sudo apt install gnupg
echo "deb [trusted=yes] http://repo.mongodb.org/apt/debian bookworm/mongodb-org/7.0 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list
sudo apt update
sudo apt install -y mongodb-org
sudo systemctl daemon-reload
sudo systemctl enable mongod.service
sudo systemctl restart mongod.service
sudo systemctl status mongod.service
sudo apt-mark hold mongodb-org

The second last line should indicate that the status is activly running. Note: If the service crashes immediatly with the error illegal instruction, check if your CPU supports the AVX flag. If you are running a Proxmox/KVM VM, the default CPU type x86-64-v2 does not support it. Change it to x86-64-v3 or x86-64-v4. If your CPU does not support both versions, you need to downgrade MongoDB to version 4.4.x, which is out of support since February 2024.

Install and configure OpenSearch

Follow these steps to install OpenSearch with all requirements.

Note: At time of writing OpenSearch 2.16 was not supported, so the install command is installing OpenSearch 2.15 instead.

sudo apt update && sudo apt -y install lsb-release ca-certificates curl gnupg2
curl -o- https://artifacts.opensearch.org/publickeys/opensearch.pgp | sudo gpg --dearmor --batch --yes -o /usr/share/keyrings/opensearch-keyring
echo "deb [signed-by=/usr/share/keyrings/opensearch-keyring] https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable main" | sudo tee /etc/apt/sources.list.d/opensearch-2.x.list
sudo apt update
sudo apt list -a opensearch
sudo OPENSEARCH_INITIAL_ADMIN_PASSWORD=$(tr -dc A-Z-a-z-0-9_@#%^-_=+ < /dev/urandom  | head -c${1:-32}) apt-get -y install opensearch=2.15.0
sudo apt-mark hold opensearch

Open the OpenSearch configuration file

sudo nano /etc/opensearch/opensearch.yml

Set the following values in the config. Uncomment them until network.hostand add the ones after to the end. This will setup an instance with minimum security, so don’t make it available to the public!

cluster.name: graylog
node.name: ${HOSTNAME}
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 0.0.0.0
discovery.type: single-node
action.auto_create_index: false
plugins.security.disabled: true

Save the Settings with Ctrl+O and leave with Ctrl+X. Edit the Java VM:

sudo nano /etc/opensearch/jvm.options

In the settings Xms1g and Xmx1g replace the 1 with the number that represents the half of your current memory (i.e Xms8g if you have 16 GB of memory). Save the Settings with Ctrl+O and leave with Ctrl+X.

Finally configure some kernel parameters and enable the service:

sudo sysctl -w vm.max_map_count=262144
sudo echo 'vm.max_map_count=262144' >> /etc/sysctl.conf

sudo systemctl daemon-reload
sudo systemctl enable opensearch.service
sudo systemctl start opensearch.service

Note: If OpenSearch does not start, check the permissions of the following folders. If only root has access, adjust the permissions with:

sudo chown opensearch:root /etc/opensearch
sudo chown opensearch:root /var/lib/opensearch
sudo chown opensearch:root /var/log/opensearch

Install and configure Graylog Open

Install the Graylog repository configuration and Graylog Open with the following commands.

wget https://packages.graylog2.org/repo/packages/graylog-6.0-repository_latest.deb
sudo dpkg -i graylog-6.0-repository_latest.deb
sudo apt update
sudo apt install graylog-server

Avoid unwanted major updates:

sudo apt-mark hold graylog-server

Now Graylog needs to be configured. Run this command to generate a password_secret we will need in the config file after:

< /dev/urandom tr -dc A-Z-a-z-0-9 | head -c${1:-96};echo;

Run this command to generate a root_password_sha2 where you enter the root password of your user and the output will be the SHA2 hash. ATTENTION! The input will be in cleartext!

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

Note the two output strings and edit the config:

sudo nano /etc/graylog/server/server.conf

Go to the password_secret and the root_password_sha2 value and set the value to the string generated before. Search for http_bind_address and sett the value to 0.0.0.0:9000. Now set the elasticsearch_hosts setting to point Graylog to the OpenSearch instance like http://127.0.0.1:9200 Save the Settings with Ctrl+O and leave with Ctrl+X.

Finally setup and start the service:

sudo systemctl daemon-reload
sudo systemctl enable graylog-server.service
sudo systemctl restart mongod.service
sudo systemctl restart opensearch.service
sudo systemctl restart graylog-server.service
sudo systemctl --type=service --state=active | grep graylog

Access Graylog

Go to the Graylog Frontend and login: http://yourserver:9000 Login with the username admin and the password you entered when you stated the root_password_sha2 Hash for the config file.

Hardening

This are just some recommendations and is not containing a guide to implement any of the hardening recommendations.

HTTPS / Reverse Proxy

One of most important thing is that you should NEVER expose Graylog directly to the public or wider internal audience. Always put a Reverse Proxy like Nginx or Apache in front of the Graylog instance. Additionally you can also configure using HTTPS by using Let’s Encrypt certificates. That way the public traffic would be encrypted and your Graylog server is not exposed to the public directly.

Firewall

Of cause a network and/or host firewall can at least block unwanted traffic, that may reach any node of the Graylog infrastructure. In case of a single node, you will only need port 9000 from the reverse proxy (or a very few internal machines) to the Graylog server and port 22 for maintenance reasons. All other incomming ports can be blocked.

http_bind_address value

If you have a static bind address and you know the IPs you get connections from, you can set this setting the the involved IP addresses. This will automatically block everyone, that should not contact the system directly.

OpenSearch hardening

There are many things that can be hardened at OpenSearch. There is a separate article at the OpenSearch page about this topic which contains encryption, authentication, access control, logging and more. See About Security - OpenSearch Documentation

Additional Infos

The following information are additional remarks for special purposes. It is not about the general setup of Graylog but may be helpful for someone.

Graylog + NLog

If you are developing apps using the Dotnet Framework and you are also using NLog for logging, this may be interesting for you. You can send logs directly from NLog to Graylog. This are the steps to do in Graylog:

In Graylog go to System>Inputs
Select GELF HTTP from the Combobox and click on Launch new input
Give the Input a title like GELF HTTP. You can type whatever you want.
Scroll to the bottom and click Launch input

In you program solution you need to do the following steps:

Add the Nuget package NLog.Gelf (yes it is old, but still works) and of cause the NLog package
If not exist, create a file called NLog.config in your project root.
Set the NLog.config to Build Action = Content in the file properties pane and set the Copy to Output directory value to Copy if newer or Copy always.
If you didn’t have NLog.config, paste the following in the new file:

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        autoReload="true"
        internalLogLevel="info"
        internalLogFile="internal-nlog.txt"
        >
        <extensions>
                <add assembly="NLog.Gelf" />
        </extensions>

        <!-- 
        See https://github.com/nlog/nlog/wiki/Configuration-file 
        for information on customizing logging rules and outputs.
        -->
        <!-- Enable async for all targets. Maximum queue limit: 10000. After that messages will be discarded-->
        <targets async="true">
                <!-- add your targets here -->
                <target xsi:type="File"
                        name="f"
                        fileName="${basedir}/logs/${shortdate}-${processid}.log"
                        layout="${longdate}|${uppercase:${level}}|${logger}|${message}|${exception:format=tostring}"
                        />

                <target xsi:type="Console"
                        name="c"
                        layout="${longdate}::${uppercase:${level}}::${logger}::${message}::${exception:format=tostring}"
                        />

                <target xsi:type="Debugger"
                        name="d"
                        layout="${longdate}::${uppercase:${level}}::${logger}::${message}::${exception:format=tostring}"
                        />

                <target type="GelfHttp"
                        name="g"
                        layout="${longdate}|${uppercase:${level}}|${logger}|${message}|${exception:format=tostring}"
                        serverUrl="http://<your Graylog Server>:12201/gelf"
                        facility="MyApp" 
                        />
        </targets>

        <rules>
                <!-- add your logging rules here -->
                <logger name="*" minlevel="Debug" writeTo="f" />
                <logger name="*" minlevel="Trace" writeTo="d" />
                <logger name="*" minlevel="Debug" writeTo="c" />
                <logger name="*" minlevel="Trace" writeTo="g" />
        </rules>
</nlog>

If you already had a NLog.config, take a look at the following settings in the example above and adjust your config accordingly:
1. The <extensions> part right after the nlog tag at the beginning
2. The async="true" property of the targets tag
3. The target tag and its properties with the type GelfHttp
4. The logger at the rules with the writeTo="g".
Save the config and (re)start your application. If your network firewall settings are correct, you should see the configured logs in Graylog.
For production purpose I recommend not to set the minlevel parameter for the Graylog logger to Trace but Info to avoid spamming the Graylog server and transfer only relevant logs to the network. Of cause temporary for testing and troubleshooting higher levels are OK.
I also recommend to use a DNS alias for the Graylog server target URL in the config instead of IP addresses or hostnames to have no static hard coded values your config and with it in the source code repository.

References

Official Installation guide: Debian Installation (graylog.org) Hardening for OpenSearch: About Security - OpenSearch Documentation

Graylog Setup