Automating detection with Notebook

Introduction

Jupyter Notebook has become a go-to tool when I need to perform analysis on attacks when raw data (eg: application logs, web logs) is provided instead of well-formatted data ingested into a SIEM. I start to think about how I could make this process repeatable so that when there is a future analysis on the same type of dataset it could be repeatable in a scalable setup.

Design

  • Rules
execution_from_download.json (an alerting rule example)
file_creation_in_download.json (an hunting rule example)
  • Alert Notebook (alert.ipynb)
alert.ipynb (notebook to run all the alert)
  • Rule: Class to define the rule object and some of the methods.
  • Detection: Class to run the rules and hunt queries.
  • Alerting: Class to perform alerting, which I’m using slack in this setup.
  • Papermill (Python script)
main.py
  • Slack
slack alert

Conclusion

The idea shared in this post is not new. In fact, any SIEM on the market can do it. However, when an analyst/investigator is tasked to investigate an intrusion with raw data provided, this could be a good way to manage their investigation workflow. By leveraging elastic and papermill, this becomes a scalable and repeatable detection platform.

Reference

https://pbpython.com/papermil-rclone-report-1.html
https://papermill.readthedocs.io/en/latest/
https://stackoverflow.com/questions/46771268/cannot-add-python3-kernel-to-jupyter
https://mordordatasets.com/introduction.html
https://api.slack.com/messaging/sending
https://github.com/Neo23x0/sigma
https://uncoder.io/
https://airflow.apache.org/
https://medium.com/@jaydenzheng/build-alerting-pipeline-with-jupyter-spark-and-sigma-11083caa739b

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store