A hands-on cybersecurity lab showing how I configured Splunk, Snort, and Kali Linux to simulate attacks, validate detections, and build a practical incident response workflow.

Introduction

Cybersecurity is not only about knowing what an attack looks like in theory. A strong security team also needs to understand how attacks appear in logs, how alerts are generated, and how an Incident Response (IR) process should move from preparation to recovery.

This project was completed as a group incident-response lab, but I took primary technical ownership of the implementation. I was responsible for building and troubleshooting most of the lab environment, configuring the SIEM pipeline, validating attack simulations, and ensuring the detection workflow worked end-to-end.

The scenario was based on a fictional company called CSA271.com. The company wanted to improve its readiness against cyberattacks, especially attacks targeting its web-facing infrastructure. Instead of only writing an Incident Response Plan on paper, I wanted to build a working lab where logs, alerts, attack traffic, and response procedures could be tested together.

The final environment used Splunk Enterprise, Splunk Universal Forwarder, Snort IDS, Kali Linux, and Windows Server systems to simulate a small enterprise monitoring setup. The project focused on detecting several common indicators of compromise, including brute-force attacks, SYN scans, TCP scans, and UDP scans.

This article explains the full process: how the lab was designed, how the SIEM was configured, how attacks were simulated, how detections were validated, what problems appeared during troubleshooting, and what I learned from building the environment.

Why I Built This Lab

A common challenge in cybersecurity learning is that many concepts sound clear in theory but become much more complicated during implementation. For example, it is easy to say that a SIEM should detect brute-force attempts or port scanning. However, in a real setup, several practical questions appear:

This lab gave me a chance to connect security theory with hands-on implementation. It also helped me practice blue-team thinking: not only how an attacker runs tools such as Hydra or Nmap, but also how those actions appear from the defender’s side.

From a career perspective, this project was valuable because it combined several practical skills used in SOC, cybersecurity analyst, and incident response roles:

Lab Environment Overview

The lab was built in a virtualized environment using Oracle VirtualBox during different stages of testing. All systems were placed in the same internal subnet so that traffic could be generated, forwarded, collected, and analyzed in a controlled network.

Network Topology

The lab used the subnet:

10.10.1.0/24

The main systems were:

System                Role                                      IP Address
--------------------  ----------------------------------------  --------------------
Windows Server 2016   Splunk Enterprise SIEM Server             10.10.1.130
Windows Server 2012   Splunk Universal Forwarder / IIS / FTP    10.10.1.129
Kali Linux            Attack simulation machine                 10.10.1.128
Web Server            Target system                             Internal lab network
FTP Server            Target service for brute-force testing    10.10.1.129

The Windows Server 2016 machine acted as the central SIEM server. The Windows Server 2012 machine hosted the Splunk Universal Forwarder and also provided IIS and FTP services. Kali Linux was used to generate controlled attack traffic.

The goal was to create a small but realistic security monitoring pipeline:

Attack Traffic → Target Server Logs / Snort IDS → Splunk Forwarder → Splunk Enterprise → Search / Alert / Investigation

Why Time Synchronization Was Important

One detail that became important early in the project was time synchronization.

In incident response, timestamps are critical. If logs from different systems do not share the same time reference, it becomes difficult to reconstruct the attack timeline. For example, a failed login event may appear before or after a related IDS alert simply because two systems have different clocks.

To address this problem, I used NTP synchronization to ensure the servers produced consistent timestamps. This made it easier to correlate events between Windows logs, Splunk data, and Snort alerts.

This was a good reminder that incident response does not start when an alert appears. It starts during preparation, when logging, time synchronization, and visibility are designed properly.

Choosing the SIEM: Splunk Enterprise

For the SIEM platform, we selected Splunk Enterprise.

The main reasons were:

Splunk Enterprise was installed on the Windows Server 2016 machine. Splunk Universal Forwarder was installed on the Windows Server 2012 machine to forward logs back to the central Splunk server.

We also used Snort IDS to improve network traffic visibility. Splunk was responsible for centralized log search and analysis, while Snort helped detect scan-related traffic patterns that were not always obvious from host logs alone.

SIEM Architecture

The SIEM architecture followed a simple enterprise-style model:

Kali Linux Attack Machine
        |
        | Simulated attacks
        v
Windows Server 2012 / IIS / FTP Services
        |
        | Logs forwarded by Splunk Universal Forwarder
        v
Windows Server 2016 Splunk Enterprise SIEM
        |
        | Search, correlation, alerting, and investigation
        v
Security Analyst / Incident Responder

Snort added another layer of network visibility:

Network Traffic → Snort IDS → Snort Logs → Splunk Index → Detection Rules / Alerts

This architecture allowed the lab to support both host-based and network-based detection. Host logs were useful for authentication activity, while Snort was helpful for scan detection.

Splunk and Forwarder Configuration

The SIEM server was installed on:

10.10.1.130

The Universal Forwarder was installed on:

10.10.1.129

The high-level setup process was:

  1. Install Splunk Enterprise on the SIEM server.
  2. Install Splunk Universal Forwarder on the forwarding server.
  3. Open port 9997 on the Splunk server for data forwarding.
  4. Configure the forwarder to send logs to the Splunk server.
  5. Configure Splunk inputs and indexes.
  6. Configure Snort log ingestion.
  7. Restart the services and verify that logs are being received.

The key Splunk configuration files included:

These files controlled which logs were collected, where they were sent, how they were parsed, and how they were indexed.

One of the biggest lessons from this stage was that SIEM implementation is not just a software installation task. The real value comes from the data pipeline: collection, forwarding, parsing, indexing, searching, and alerting.

If any part of that pipeline is broken, the SIEM may appear to be running, but it will not provide useful detection.

Services Used for Testing

To generate meaningful security events, I configured target services on the Windows Server 2012 machine.

The main services included:

The FTP service was especially useful because it enabled simulating brute-force login attempts with Hydra and Medusa on Kali Linux.

This also reinforced a basic security principle: every exposed service increases the attack surface. If a service must be exposed, it should be monitored properly.

Attack Simulation and Detection

After configuring the SIEM and forwarding components, I moved into the attack simulation phase. The purpose was not to compromise a real system, but to validate whether the monitoring pipeline and IR plan could detect suspicious behaviour.

The lab focused on four attack categories:

  1. Brute-force attacks
  2. SYN scans
  3. TCP scans
  4. UDP scans

Brute-Force Attack Simulation

A brute-force attack attempts to guess valid credentials by repeatedly trying username-password combinations. In real environments, this type of attack can target SSH, FTP, VPN portals, web login pages, or remote access services.

In this lab, I used Hydra to test FTP and SSH login attempts:

hydra -l admin -P /usr/share/wordlists/rockyou.txt ftp://10.10.1.129 -t 1 -f -V
hydra -l admin -P /usr/share/wordlists/rockyou.txt ssh://10.10.1.129 -t 1 -f -V

The detection idea was to look for repeated failed authentication attempts from the same source within a short time window.

A practical detection rule should consider:

From an incident response perspective, brute-force detection should not stop at counting failed logins. An analyst should also check whether the attacker eventually succeeded.

Possible response actions include:

SYN Scan Detection

A SYN scan is commonly used during reconnaissance. It sends SYN packets to target ports to determine whether ports are open, closed, or filtered. Since it may not complete the full TCP handshake, it is often referred to as a half-open scan.

I simulated a SYN scan with Nmap:

nmap -sS 10.10.1.129

The detection strategy involved using Snort to identify excessive SYN packets and then forwarding those alerts into Splunk.

Signs of a SYN scan may include:

A SYN scan does not always mean that a system has been compromised. However, it can be an early sign that an attacker is mapping the environment before launching a more targeted attack.

TCP Scan Detection

A TCP connect scan completes the full TCP handshake when checking ports. It is usually easier to detect than a SYN scan because it creates more complete connection records.

I simulated a TCP scan using:

nmap -sT 10.10.1.129

The detection logic focused on identifying unusual connection attempts across multiple ports.

Useful indicators include:

In Splunk, this can be detected by grouping events by source IP and counting unique destination ports. If a single source connects to many ports in a short period, it may indicate scanning behaviour.

UDP Scan Detection

UDP scanning was one of the more difficult parts of the project.

Unlike TCP, UDP is connectionless. There is no handshake, and some services may not respond at all. Because of this, UDP scans can be harder to detect and interpret.

I simulated UDP scanning with:

nmap -sU 10.10.1.129

Possible signs of UDP scanning include:

This part of the lab showed why network-based visibility is important. Host logs alone may not provide enough information for certain types of reconnaissance. Snort helped fill that gap by detecting network-level patterns and producing alerts that could be reviewed in Splunk.

The Most Memorable Debugging Issue: Snort Configuration and Misleading Error Logs

The most memorable challenge in this project came from Snort.

At one point, Snort was not producing the expected detection results. The issue was related to the configuration file. Since Snort can be deployed in different environments, the configuration details for Windows and Linux are not always identical. Some paths, formatting expectations, and configuration assumptions can behave differently across platforms.

What made this issue difficult was that the error message did not clearly point to the actual problem. Instead of showing the exact configuration line or setting that caused the failure, the error appeared in a confusing location. My impression was that the problem may have occurred when Snort was reading or parsing the configuration file, causing the error to appear as a more general formatting or loading issue.

This forced me to debug the issue step by step rather than relying solely on the error message. I had to check whether Snort was reading the correct configuration file, whether the syntax matched the Windows environment, whether the rule paths and log paths were correct, whether the output format was usable, and whether Splunk could ingest the generated logs.

The troubleshooting process looked like this:

Is Snort loading the correct configuration file?
Are the Windows paths written in the expected format?
Are the rule files being loaded correctly?
Is Snort actually seeing the network traffic?
Are alerts being generated?
Are logs being written to the expected location?
Can Splunk read and index those logs?
Can I search and validate the events in Splunk?

This was frustrating at the time, but it became one of the most valuable learning moments in the project. It taught me that in security engineering, an error message does not always directly identify the root cause. Sometimes the real issue is one layer earlier in the pipeline, such as configuration parsing, file paths, platform differences, or log output format.

The experience also reinforced an important detection engineering lesson: a tool can appear to be installed and running, but still fail to provide useful detection if the configuration, logging, or ingestion pipeline is not validated end-to-end.

Detection Workflow

The general detection workflow followed this process:

  1. Generate controlled attack traffic from Kali Linux.
  2. Confirm that the target system or Snort generated logs.
  3. Forward the logs to Splunk.
  4. Search and validate the relevant events.
  5. Build detection logic around the event patterns.
  6. Trigger alerts based on thresholds or suspicious behaviour.
  7. Document the incident response steps.

This workflow helped connect the lab's offensive and defensive sides. Running Nmap or Hydra was only one side of the exercise. The more important part was understanding what defenders can actually see and how they should respond.

A major takeaway was that detection rules must be validated. It is not enough to assume that a SIEM or IDS will automatically detect an attack. The analyst needs to confirm that logs are generated, forwarded, parsed, indexed, and searchable.

Incident Response Plan

The technical lab was connected to a structured Incident Response Plan. The response lifecycle included preparation, detection, analysis, containment, eradication, recovery, and lessons learned.

1. Preparation

Preparation included configuring logging, deploying Splunk, installing the Universal Forwarder, enabling Snort, opening required ports, and synchronizing system time.

This stage matters because incident response depends on visibility. If logging is missing or misconfigured, analysts may lack sufficient evidence to understand what happened.

2. Detection

Detection involved monitoring Splunk alerts and reviewing suspicious events. Examples included repeated failed logins, excessive SYN packets, many TCP connection attempts, and suspicious UDP traffic.

The goal was to identify suspicious behaviour as early as possible.

3. Analysis

During analysis, the security analyst investigates whether the alert is a true positive, which system was targeted, which source IP address was involved, and whether the activity was successful.

For example, after a brute-force alert, the analyst should check whether a successful login occurred after repeated failures.

4. Containment

Containment actions may include:

The goal is to stop the attack from continuing or spreading.

5. Eradication

Eradication focuses on removing the root cause and any malicious artifacts. In a real environment, this may include removing malware, patching vulnerabilities, closing unnecessary ports, or hardening weak credentials.

6. Recovery

Recovery ensures that affected systems return to normal operation safely. After recovery, systems should continue to be monitored to confirm that the attacker has not returned.

7. Lessons Learned

After the incident, the team should document what happened, what worked, what failed, and what should be improved. This stage is important because every incident should improve the organization’s future security posture.

What I Learned

This project gave me a much deeper understanding of how detection and incident response work in practice.

The most important lessons were:

The Snort debugging issue was especially valuable because it reminded me that tools do not magically solve security problems. Analysts and engineers still need to understand the data flow, validate assumptions, and confirm that alerts represent real activity.

Recommendations for Future Improvements

If I were expanding this project further, I would improve it in several ways.

Add More Log Sources

Additional firewall, endpoint, web application, and Active Directory logs would make the lab more realistic.

Build More Advanced Correlation Rules

Instead of detecting each activity separately, Splunk could correlate multiple events together. For example, a SYN scan followed by FTP brute-force attempts from the same IP should be treated as a higher-priority incident.

Add Dashboards for Analyst Review

A dashboard could show:

This would make the investigation faster and more organized.

Add Automated Response

For high-confidence alerts, automated response actions could be tested. For example, repeated brute-force attempts from a single IP address could trigger a temporary firewall block.

Test More Attack Scenarios

Future testing could include web attacks, privilege escalation, malware simulation, DNS tunnelling, data exfiltration, and lateral movement.

Conclusion

This project was a practical exercise in building a SIEM-based incident response lab from the ground up.

Although it was completed as a group project, I took the main technical ownership of the environment and gained hands-on experience configuring Splunk, forwarding logs, integrating Snort, simulating attacks, and validating detections.

The lab helped me understand that cybersecurity detection is not just about having the right tools. It is about making sure the tools are configured correctly, the logs are reliable, the alerts are meaningful, and the response process is clear.

For anyone pursuing a cybersecurity analyst, SOC analyst, or incident response role, building a lab like this is a valuable way to connect theory with real operational skills.

The most important lesson I took away from this project was simple:

Detection is only useful when it is tested, validated, and connected to an actionable response plan.

Building a SIEM-Based Incident Response Lab with Splunk, Snort, Kali Linux, and Windows Server was originally published in System Weakness on Medium, where people are continuing the conversation by highlighting and responding to this story.