Automating detection with Notebook
The opinions expressed in this post are my own and not necessarily those of my employer
Jupyter Notebook has become a go-to tool when I need to perform analysis on attacks when raw data (eg: application logs, web logs) is provided instead of well-formatted data ingested into a SIEM. I start to think about how I could make this process repeatable so that when there is a future analysis on the same type of dataset it could be repeatable in a scalable setup.
In this post, I’m going to discuss how to use Papermill, a python library to automate detection.
The design is simple. An analyst can write notebooks for a hunt or test out new alert rule.
Rule repo will contain both rules for alerting and hunting. Alerting rules are rules that have high fidelity while Hunting rules are rules that have low fidelity and not suitable for alerting purposes.
The alert.ipynb notebooks will read rules from the Rule Repo, which are rules written in elastic query language and query against the elastic cluster. If there are any matches, it will perform alerting in slack.
I’m using elastic for the datastore since it is free and scalable. However, you could use any datastore(eg: flat file, any database) or any alerting (eg: slack, thehive) as long as you build the connectors to communicate with the different datastore and alerting.
The rule file contains the elastic search query and also the filter key to remove any known false positive. The idea is that you can build up your queries and whenever you are into a new investigation, you can run through the queries and see if there are any matches.
You should also check the Sigma project and uncoder.io for rules creation.
- Alert Notebook (alert.ipynb)
Since we are using papermill to run the alerting.ipnyb notebook, we will only require 1 notebook to run all the rules as papermill allows us to parse value into notebook during run time.
There are 3 classes imported in the notebook:
- Rule: Class to define the rule object and some of the methods.
- Detection: Class to run the rules and hunt queries.
- Alerting: Class to perform alerting, which I’m using slack in this setup.
The rule_name is a variable created to receive input when papermill executes. We will parse the rule filename into this variable in order to execute the rule in the alerting notebook.
This notebook is really straight forward. In real life, you will probably want to include enrichments in the notebooks as well. Refer to my previous post on how you can perform enrichment with Cortex.
- Papermill (Python script)
The script to run the notebook with papermill is straight forward. The get_rules() will read all the rule files in the rules directory. And the main() will loop through the rule list and parse the file name of the rule to the notebook and the notebook will run the cells with the rule accordingly.
If there is any hit for the query, it will trigger a slack alert with the Alerting class. However, as mentioned earlier, you could also create a thehive connector to open a hive ticket instead of slack.
P/S: I’m using APT 29 data set from the Mordor project for the blog post.
The idea shared in this post is not new. In fact, any SIEM on the market can do it. However, when an analyst/investigator is tasked to investigate an intrusion with raw data provided, this could be a good way to manage their investigation workflow. By leveraging elastic and papermill, this becomes a scalable and repeatable detection platform.
There are definitely many improvements that can be made to the design. For example, Airflow could be included to run the notebooks instead of a simple python script as Airflow has much better scheduling capability, scalability, and many other useful features. This will be a better option if the plan is to run this detection with real-time ingestion into the datastore. Also, working on the filtering capability and chaining up different rules for better alerting.
Thanks for reading and feel free to reach out if you find this useful/interesting!