In short: The IoC Lookup downloads IoCs (“Indicator of Compromise”, i.e., malicious domains, IP-addresses, or URLs) from different blocklists and unites them into three csv files (one for each type). The information of this files is served via a REST-API and can be used by Graylog through an HTTP-JSONPath Adapter to serve a Lookup Table.

A common usage scenario is to tag and enrich all kind of logs (i.e., Active Directory, VPN, DNS, Email, etc.) containing malicious IPs, domains or URLs with detailed information about the IoC for further investigation.

It requires little work to set up, the most extensive being configuring the Flask Server for the API and setting up the JSON file with the blocklists to download.

Furthermore, it is easily expandable for new blocklists or Use Cases.

IOC-Lookup 1.0 für Graylog

This plugin reads CSV-Lists with IOCs via HTTP-JSON Path-Adapter.

Cost: 250,- Euro excl. Vat

Contact Sales

Initial Installation of IoC Lookup

After you receive your copy of IoC Lookup copy it to a system running any Linux operating system featuring systemd. Now you need to install the Python dependencies. You can do this by executing the following command inside the IoC Lookup folder: “pip install -r requirements.txt”. If you have a system with booth Python 2 and Python 2 installed, you need to call “pip3” instead of “pip”. Afterwards you need to create two folders inside your program directory: “downloads” and “lists”.

For the Flask Server to work you need to install “gunicorn”, this can be done with the console command: “sudo apt install gunicorn3”. If you wish you can use “apt-get” instead of “apt”. Additionally, you can install “gunicorn” using pip: “pip install gunicorn”.

Configuration of ioc-lists.py

Configuration of the main program (“ioc-lists.py”) is done via the “ioc-lists.conf” file. This configuration file contains all information needed for the program to run. You need do adjust lines 3 and 6 of the file to contain the absolute path to the two directories.

Initial configuration of blocklists to download

IoC Lookup is delivered with a preselection of different blocklists to use. If you want to include other blocklists see “How to customize”.

Configuration of ioc-server.py

Configuration of the Flask server relies on “ioc-lists.conf” and “ioc-server.service”. Furthermore, you need to create a file “token.txt” in your program directory. The configuration file “ioc-lists.conf” was already configured in a previous step, so there is no action required.

However “ioc-server.service” needs additional configuration. In line 7 you need to provide the absolute path to “ioc-lists.conf” and in line 8 the absolute path to the newly created file “token.txt”. In lines 9 and 10 you need to set the user and group which should run the server. Additionally, you need to adjust line 15 to match the absolute path of the working directory. By default, the server runs on port 5000, if you want to change the port adjust line 16. Depending on the installation directory of “gunicorn” you also need to adjust its directory in line 16.

Finally, you need to edit “token.txt”. This file contains the authorization token required for the API endpoints. Simply create a new and secure token and paste it into this file.

Daemon for Flask server

1. Place the file “ioc-server.service” in the directory “/etc/systemd/system”
2. Start the server with “systemctl start ioc-server”
3. Enable automatic start on boot: “systemctl enable ioc-server”

Cron Job

The creation of a cron job is recommended to regularly execute IoC Lookup and update its lists and the server. The execution interval depends on your conditions.

The crontab for IoC lookup should follow the following scheme:

python3 <absolute-path>/ioc-lists.py –config <absolute-path>/ioc-lists.conf && curl -XPOST -s –show-error “http://<ip:port>/reload?token=<token>” > /dev/null

For example:

(click to enlarge)

With this command the IoC Lookup gets updated every hour and automatically reloads the Flask server.

Graylog Lookup Table creation

To utilize the IoC Lookup in Graylog you need to create a Lookup Table. A Lookup Table in Graylog consists of three parts: the Lookup Table, a Data Adapter, and a Cache.

First you will create the Data Adapter. To do so navigate to the Lookup Table page in Graylog via the menu System->Lookup Tables and then select Data Adapters from the top right menu. After that you click on Create data adapter and select HTTP JSONPath as the Data Adapter. Fill out the form as follows:

1. Title: Enter “Adapter: IoC Lookup” as the name
2. Lookup Url: http://<ip:port>/lookup?token=<token>&value=${key} Replace <ip:port> and <token> with your corresponding values.
3. Single value JSONPath: $.ioc
4. HTTP User-Agent: Graylog Lookup – https://www.graylog.org/

It should look something like this:

(click to enlarge)

You can save the Data Adapter now and proceed with the creation of the Cache. Select Cache from the top right menu and Create Cache. From the dropdown select “Node-local, in-memory cache”.

Configure the adapter as follows:

1. Title: “Cache: IoC Cache”
2. Maximum Entries: 1000000
3. Expire after access: tick the checkbox and select 45 seconds

It should look like this:

(click to enlarge)

Finally, you configure the Lookup Table itself. Select Lookup Tables from the top right menu and then Create lookup table. Give the Lookup Table a name like “ioc-lookup” and select the previous created Data Adapter and Cache. Save the Lookup Table and it’s ready to use.

Graylog Pipeline creation

To use the created Lookup Table, you need to use Graylog Pipelines. For IoC Lookup at least one Pipeline is required. To create the rule, navigate to Pipelines via the menu System->Pipelines and create a new rule. Give the rule a description and enter following code in the section “Rule source”:

rule “ioc_lookup”
when
  has_field(“ioc_lookup_value”)
then
  set_field(“ioc_lookup_result”,
  to_string(lookup_value(“ioc-lookup”,
  $message.ioc_lookup_value)));
end

It should look like this:

(click to enlarge)

The rule requires the field “ioc_lookup_value” to exist and contain the value (IP, domain or URL) to look up. If this field exists it makes a call to the Lookup Table and writes the result in a newly created field named “ioc_lookup_result”. The created rule needs to be attached to a Pipeline. The Pipeline to which it gets attached depends on your needs. It can be used with a variety of logs, they only need to contain a IP, domain or URL.

How the field “ioc_lookup_value” is created and populated depends on your Graylog setup. For example, it can be done with other Pipelines and isn’t part of this guide.

How to customize

IoC Lookup is highly customizable regarding the used blocklists. It ships with a preselection of blocklists and parsers for them.

Adding a new blocklist

If you need or want to add other blocklists to monitor more Use Cases you can add the new blocklist to “downloads.json” according to the following format:

{
  “url”: <Full URL to the blocklist>,
  “source”: <Source of the blocklist>,
  “type”: <Type of file (“text/plain” or “text/csv”)>,
  “content”: <Content of file (ip/domain/url)>,
  “parser”: <Parser to be used>
}

You can look at the predefined blocklists for further guidance and for information about the existing parsers.

Adding a new parser

For parsing the blocklists an abstract factory pattern (see https://en.wikipedia.org/wiki/Abstract_factory_pattern for more information) is used. All code regarding the parsers can be found in the folder “ioc”. The folder “ioc” contains a folder “parsers” which contains subfolders for the supported file types (text, csv, json). The structure of relevant folders/files is as following:

• ioc/parsers
  • _init_.py
  • parser_factory.py
  • csv
    • _init_.py
    • factory.py
    • parser.py
  • json
    • _init_.py
    • factory.py
    • parser.py
  • text
    • _init_.py
    • factory.py
    • parser.py

Writing a new parser for an existing file type

If you need a new parser for an existing file type, you need to edit the files “factory.py” and “parser.py” inside the corresponding subfolder.

1. First step is to write a create a new class inside the “parser.py” file.
a. The class needs to implement the static method “parse(text_file, source)” which returns the IoC data.

2. Afterwards you need to edit the file “factory.py” to register the new class.
a. Add a new “elif”-case for the parser and call the “parse” function of the parser.

For further guidance look at the other parsers’ implementations and how they are registered in both files.

Writing a parser for a new file type

If you need to parse a new file type you need to add a new subfolder for that type and create three files “__init__.py”, “factory.py” and “parser.py” inside this subfolder.

1. 1. First step is to create the needed parser inside “parser.py” (see description in Writing a new parser for an existing file type).
   2. After that you need to import the parser in the file “factory.py”, this can be done with “from .parser import *”. This imports all parsers from their file.
   3. Now you need to create the factory class for that file type in “factory.py” and define the static method “create_<file-type>_parser(file)”. Substitute <file-type> with your corresponding file type. This function is used to call the parsers of the file type.
   4. Export the newly created factory in the “__init__.py” file. This can be done with “from .factory import <factory-name>”.
   5. Import the new factory in “parser_factory.py”. Look at how the other factories are imported for guidance.
   6. Register the new factory and file type in the method “create_parser(file)” of the file “parser_factory.py”.

For more guidance and information look at how the other factories and parsers are implemented and repeat the same process.