In short: The Context Collector unifies information scattered over different messages, identified by key-values, into one new singular message, enabling easier filtering/search and giving analyst a fast overview over multiline logs.
Common scenarios are the unification of multi-line messages such as E-Mail-Server logs, Session Attributions for Cisco ISE and Cisco ASA VPNs or Special Privilege Logon Matching in Active Directory Audit Logs. It does not require additional infrastructure to run and is configurable via the Graylog WebUI. The context collector also supports cluster-installations, where relevant messages might arrive at different nodes.
Context Collector for Graylog V4.0
This plugin for Graylog collects multiple fields from different messages with a common key or ID.
Cost: it’s free!
IMPORTANT: The Context Collector can be downloaded here free of charge for your own use. Distribution by others other than NetUSE AG is expressly prohibited!
Context Collector for Graylog V4.1
This plugin for Graylog collects multiple fields from different messages with a common key or ID.
Cost: it’s free!
IMPORTANT: The Context Collector can be downloaded here free of charge for your own use. Distribution by others other than NetUSE AG is expressly prohibited!
Context Collector for Graylog V4.2
This plugin for Graylog collects multiple fields from different messages with a common key or ID.
Cost: it’s free!
IMPORTANT: The Context Collector can be downloaded here free of charge for your own use. Distribution by others other than NetUSE AG is expressly prohibited!
Context Collector for Graylog V4.3
This plugin for Graylog collects multiple fields from different messages with a common key or ID.
Cost: it’s free!
IMPORTANT: The Context Collector can be downloaded here free of charge for your own use. Distribution by others other than NetUSE AG is expressly prohibited!
How does it work
The plugin is used by configuring so called “Collections”. Each collection is made up in its core from these configuration settings:
- List of key field names
- List of value field names
- Timeout in seconds
When a message that contains all the key fields is processed, a new collection item is created. It is identified by the values of those key fields. From this and all subsequent matching messages, the fields matching a value field are added to the collection item up until all fields are present (collected) or there hasn’t been an update in the interval specified by the timeout. Then a new message is created containing the collected fields and meta-data about this specific collection.
The plugin adds a new input and output type to Graylog. Messages for processing are routed to the Context Collector by attaching the output to streams. The input is used to send newly created messages to Graylog.
Use Cases
2.1 E-Mail logs from sendmail or postfix
E-Mail-Server setups are notorious for scattering relevant information across log messages as several different components handle an E-Mail while it traverses the system. Thankfully these components often use a queue id or the RFC5322 Message-ID to attribute log messages to specific E-Mails.
When troubleshooting issues with the mail service, it is easy to filter the relevant log messages once the relevant IDs have been identified. The identification of relevant IDs is usually the time intensive part.
With the Context Collector we can collect relevant information for E-Mails based on those IDs into one message. Making it easier to filter for and identify messages based on troubleshooting requests such as “Emails from X to Y this morning”, “All messages involving relay X” or “All messages to X that had a spam score higher than Y last week”.
2.2 Active Directory: Match privileged logon to system
When an account is logged in on a system, a corresponding event ( 4624 – An account was successfully logged on) is generated. If this log on holds sensitive privileges such as SeImpersonate an additional event (4672 -Special privileges assigned to new logon) is generated.
While this event is by default quite noisy, as it is generated by NT/SYSTEM all the time, it can be filtered down to non-systemic accounts, which gives a good overview over privileged account activity. However, this event does not contain the system it originated on, which is important information for an analyst to reason about the legitimacy of an activity. This information is retained in event 4624. With the session ID it is possible to combine the information into one tuple (Account name, Special Privilege, System) with the Context Collector.
2.3 Cisco ISE
The Cisco Identity Services Engine (ISE) enables a dynamic and automated approach to policy enforcement. This approach simplifies reliable and secure control of network access. ISE promotes software-defined access and automates network segmentation in IT and OT environments.
During authentication the Cisco Identity Services Engine creates several log entries over which important key data for debugging is spread. This includes information such as assigned IPv4/6 addresses, MAC address, user authentication name, assigned policy and matched rule. By using the session ID this can be unified into one log message with the Context Collector.
The following fields are relevant to the evaluation:
type, name, timeout, key-fields, drop_incomplete, enabled, value_fields
2.4 Cisco ASA
With ID 734003 the Cisco ASA firewall solution generates logs for connecting AnyConnect Clients containing basic information about the client such as:
Username, client version, platform, applied policies, device type, MAC-address
Unfortunately, each information is its own log message, making it hard to compile reports on for example the distribution of versions across platforms or similar. The Context Collector can be here used to unify the information for one client into one message.
Installation
Please refer to https://docs.graylog.org/docs/plugins#installing-and-loading-plugins for the latest instructions for loading a plugin into Graylog.
3.1 Initial Configuration
For the Context Collector to work the input must be created and one output must be created and attached to at least one stream of interest.
3.2 Create the Context Collector Input
To create the input, go to System -> Inputs, select the “Context Collector Message Output” as new input and click “Launch Input”. In the configuration dialog tick the Global checkbox and give the input a name, such as “Context Collector Input”. Then create and start the input by clicking “Save”
3.3 Create the Context Collector Output
To attach the Context Collector Output to a stream the first time, go to the Streams page. For the desired stream select “More Actions” -> “Manage Output”. In the select box next to “Launch new Output” button, select the “Context Collector Output” as type. Click the button, in the configuration dialog give the output a name, such as “Context Collector Output”. Click Save to finish the setup. The output is now attached to the stream and processing messages.
3.4 Attach the Context Collector Output to an additional stream
It is not needed to create a new Output for each stream. To attach additional streams to the output, go to the output management page for the respective streams via “More Actions” -> “Manage Outputs”. In the select box titled “Select existing output” select the Context Collector Output you created in the previous step. To finish click the button labelled “Assign existing Output”.
3.5 How to configure a Context Collection
The configuration modal for the plugin can be found under System -> Configurations. Per default it is set to “Cluster Config”, which means that the configuration of the Collections is stored within Graylog. To support legacy deployments, it is possible to read the configuration from file.
Clicking the “Add Collection” button in the main configuration dialog for the plugin, creates a new empty collection. Clicking on the “Edit” Button in the row opens the configuration dialog for the collection. Here we can choose a proper name, the fields we want to match and collect, and the timeout. When the “Drop incomplete Collections on timeout” checkbox is ticked, collection items that did not collect all the fields specified will be discarded on timeout. There is no new message created for them.
Clicking the confirmation button brings us back to the main configuration dialog for the plugin. To save and automatically apply our changes to the processing we click the Save button.
FAQ
Question: Is the plugin compatible with Graylog Enterprise?
Answer: Yes, the messages generated will be accounted like any other message.
Question: What are the prerequisites for using the Plugin?
Answer: A Graylog installation in matching version for the plugin build. There’s no additional software needed.
Question: How are the collected messages shared between different nodes of a cluster?
Answer: For storing configuration and data the plugin uses the underlying MongoDB through Graylog facilities. You might configure the “wild tiger” to limit the MongoDB in memory-consumption.
Question: Can we run a Context Collector for one Graylog Version also on another?
Answer: Generally, not. While the processing itself usually continues to be compatible across the minor versions, the configuration tends to not render properly. We will ask you to buy a version matching your Graylog-Version.
Question: Which metadata is attached to each Context Collector message?
Answer: The following additional fields are contained in the messages:
- context-collector_complete: this shows if all requested fields are filled. It is possible to drop not complete messages directly in the Context Collector.
- context-collector_start: time when the collection was started
- context-collector_end: time when the collection collected the last message
- context-collector_name: the name of the collection. You might have multiple collections
- context-collector_timeout: if not all requested fields are found: the timestamp of the timeout.
Question: Can I use several fields as key for a collection?
Answer: Yes, while in most case only one field is needed, is it possible to use several fields as key.