Monitoring Environment
Scenario
Virtual Space Industries (VSI) believes they were recently attacked by one of the competitors. The Networking team has provided us with log files from the time they believe the attack occurred. VSI management would like us, the SOC team, to review the logs and provide them with a comprehensive analysis. This analysis will be completed using Splunk, known for its robust log management and visualizations. Additionally, we will provide recommendations for mitigations to reduce, or prevent, future attacks. During this analysis we will create reports, alerts and dashboards to be used by VSI to quickly respond to any future security incidents.
Logs Analyzed
These Windows logs contain events that occurred on the Windows server that contains VSI’s intellectual property. They are related primarily to user activity that occurred on the Windows server.
- Windows Logs
These are the web server logs that are from the webserver that contains the company’s primary public website vsi-company.com. The logs contain the information about the status of requests to and from the webserver. They also log the requestor’s IP, the requested URI and the user agent used.
- Apache Logs
Windows Logs
Reports—Windows
- Designed the following reports:
| Report Name | Report Description |
|---|---|
| Alert Signatures and Their ID | Reported alerts and their corresponding ID |
| Windows Events by Severity | Windows Events with severity, count and percentage of total. |
| Windows Events by Status | Windows by status with count and percentage of total. |
Images of Reports—Windows
- Images shown are from attack logs
Alerts—Windows
- Designed the following alerts:
| Alert Name | Alert Description | Alert Baseline | Alert Threshold |
|---|---|---|---|
| Windows activity failed events exceeded | Number of Windows failed events in 1 hour | 6 | >13 |
JUSTIFICATION: A baseline was calculated with the following Splunk search to get an average number of failures per hour. [source="windows_server_logs.csv" status=failure |stats count by status |eval average=avg(count/24) |fields average] The result was 5.9. Which was rounded up to 6. The threshold chosen because the peak on original search was 10. I doubled the baseline and added one to it to reduce false positives and alert fatigue.
- Designed the following alerts:
| Alert Name | Alert Description | Alert Baseline | Alert Threshold |
|---|---|---|---|
| Suspicious Windows Logon Activity | Suspicious number of successful Windows logons detected | 13 | >30 |
JUSTIFICATION: A similar formula as the previous slide was used to calculate the baseline to 13.4. Which was rounded to 13. The threshold was chosen because the peak on original search was 21. I doubled the baseline and added four to it to reduce false positives and alert fatigue.
- Designed the following alerts:
| Alert Name | Alert Description | Alert Baseline | Alert Threshold |
|---|---|---|---|
| Suspicious Windows User Account Deletion | Suspicious number of deleted user accounts. Based on Event ID. | 13 | >30 |
JUSTIFICATION: A similar formula as the previous slide was used to calculate the baseline to 11.2. There were some low gaps in the deletion count, so I used a divisor of 9 hours on a zoomed in section of the search results. Which was rounded to 11. The threshold was chosen because the peak on original search was 17. I doubled the baseline and added four to it to reduce false positives and alert fatigue.
Dashboards—Windows
Apache Logs
Reports—Apache
- Designed the following reports:
| Report Name | Report Description |
|---|---|
| Top HTTP Methods | Top HTTP methods in one hour |
| Top HTTP Status Codes | Top HTTP status codes in one hour |
| Apache Top 10 Referrers | Top 10 referring URI's in one hour |
Images of Reports—Apache
Alerts—Apache
- Designed the following alerts:
| Alert Name | Alert Description | Alert Baseline | Alert Threshold |
|---|---|---|---|
| HTTP Activity by Non-US Countries | HTTP Activity by Non-US Countries | 73 | >150 |
JUSTIFICATION: A similar formula as the previous slide was used to calculate the baseline to 73. There was 84 hours of data in the base report, so I used a divisor of 84 hours. The threshold was chosen because the peak on original search was 120. I doubled the baseline and added four to it to reduce false positives and alert fatigue.
- Designed the following alerts:
| Alert Name | Alert Description | Alert Baseline | Alert Threshold |
|---|---|---|---|
| Excessive HTTP POST Requests | Suspicious number of HTTP POST requests | 1 | >12 |
JUSTIFICATION: A similar formula as the previous slides was used to calculate the baseline to 1. The threshold was chosen because the peak on original search was 7. With that in mind I chose to start with 12 as a threshold to reduce false positives and alert fatigue. If there is malicious POST activity it will generally be much larger than the baseline of one. Since the high was 7 I thought it best to pick a threshold a little less than double that peak.
Dashboards—Apache
Attack Analysis
Attack Summary—Windows
- Our reports showed these findings:
- High Severity Alerts increased from 7% to 20%.
- Successful Events only increased by 1%. This did not indicate suspicious activity, even though an attack was occurring.
- Failed Events spiked at 35 at 8 AM on 3/25/20 and would have triggered the corresponding Alert.
The high number of failed events occurred over a short period time, only 13 minutes, during the 8 AM hour. By the time, the corresponding Alert would have triggered the damage was already in progress for 20 minutes. The frequency of that Alert should be increased. Once the failed events started occurring the attacker started cleaning up their tracks by deleting the audit logs. As indicated by the blank spaces that occur in the “Failed Events” report. All 3 reports allowed us to narrow down the times different portions of the attack occurred.
Our alerts showed these findings:
- There was a suspicious amount of failed activity from 8 AM to 8:41 AM on 3/25/20. The configured alert threshold of 13 would have triggered the alert.
- Our original alert configuration of the “Successful Logons” alert was not triggered during the attack. During the review of this attack, the alert was reconfigured and tested to ensure that the alert would be triggered based on the Event ID or the signature text. The threshold was not changed.
- There was no indication of a suspicious volume of deleted accounts. This alert needs to be reviewed, and the threshold changed to 13. Even though this might create false positives that have to be reviewed. This attack indicates a need for the change.
Our dashboard indicated these findings:
- 2 suspicious signatures were detected. “A user account was locked
out” & “An attempt was made to reset an accounts password”
- “A user account was locked out” from 1 AM 3 AM on 3/25/20 peaked at 896 events.
- “An attempt was made to reset an accounts password” from 9 AM to 11 AM on 3/25/20 peaked at 1258 events.
- 2 suspicious signatures were detected. “A user account was locked
out” & “An attempt was made to reset an accounts password”
3 users had abnormally high event counts.
- user_k 2,118 events or 35.603% of total events
- user_a 1,878 events or 31.568% of total events
- user_j 398 events or 6.69% of total events
user_k showed suspicious activity from approximately 9 AM to 11 AM on 3/25/20.
User_a showed suspicious activity from approximately 1 AM to 3 AM on 3/25/20.
User_j showed suspicious activity from approximately 11 AM to 1 PM 3/25/20.
- Highest peak events by minute:
- user_k peaked at 98 events (9:56 AM to 9:57 AM on 3/25/20)
- user_a peaked at 495 events (1:52 AM to 3/25/20 1:53 AM on 3/25/20)
- user_j peaked at 10 events (11:07 AM to 11:08 AM on 3/25)
- Highest peak events by minute:
The short duration times of the events would indicate the attack was likely mostly automated.
- Highest peak events by hour:
- user_k peaked at 1,256 events (9 AM to 10 AM on 3/25/20)
- user_a peaked at 984 events (2 AM to 3 AM on 3/25/20)
- user_j peaked at 196 events (11 AM to 12 PM on 3/25/20)
- Highest peak events by hour:
The long duration times and staggered occurrence indicate that the attack had multiple intentions and likely performed with different levels of security access.
Attack Summary—Apache
Our reports showed these findings:
POST requests increased to almost 30%. The POST requests are used to submit or update information to a web server.
There were no suspicious changes in the referring domains. The top domains were still a competitor, semicomplete.com.
The HTTP 404 response code increased from 2% to 15%. A likely indication of an attack.
Our alerts showed these findings:
A very suspicious amount of traffic from the Ukraine.
- Total count for all countries was 937 events from 8 PM to 9 PM on 3/25/20.
- Total count from Ukraine was 864 events from 8:05 PM to 8:06 PM on 3/25/20. The currently configured alert would be triggered by this activity.
The short duration of these event would likely indicate an automated attack.
There were a suspicious number of POST requests, 1296, from approximately 8 PM to 9 PM on 3/25/20. The currently configured threshold for this alert would have been triggered.
- Our dashboard indicated these findings:
- HTTP GET and POST methods are suspicious
- HTTP GET -6:05 PM to 6:06 PM on 3/25/20
- HTTP POST - 8:05 PM to 8:06 PM on 3/25/20
- Notice the short duration time once again. And the staggered attack times.
- During the peak time of the attack, which only lasted a minute in
these cases.
- GET peak count was 729
- POST peak count was 1296
- The Cluster map showed suspicious increased non-US traffic from the
Ukraine. Specifically, from the cities of Kiev, with 439 events, and
Kharkiv, with 432 events.
- The URI panel indicated that “/VSI_Account_logon.php” had a suspicious number of non-US requests.
- The Cluster map showed suspicious increased non-US traffic from the
Ukraine. Specifically, from the cities of Kiev, with 439 events, and
Kharkiv, with 432 events.
Summary and Future Mitigations
Project 3 Summary
There were multiple unauthorized user that accessed the Windows Systems at different periods of the day on 3/25/20. Each of the users was performing a different function likely in preparation for additional attacks later. The user accounts were user_a, user_k, and user_j.
user_a: Logged in at 12:10 AM and began assigning special elevated privileged access to multiple user accounts with the format of “user_” and computer accounts with the format of “AMCE_.” The bulk of this activity occurred between 1 AM and 3 AM on 3/25/20.
- Events related to “User account was locked out” can be seen during this time.
- user_k: Logged in at 9:13 AM on 3/25/20 and began the initial attack. This user was still logged in and active when the logs stopped.
- Events related to “An attempt to reset and accounts password” can be seen during this time.
user_j and user_n are quite possibly Service Accounts used to assign special privileges to rouge service processes. While user_n did not set off many alerts this is likely due to the privileges assigned to the account. User_j was the most active of these accounts beginning at 10:53 AM.
- Events related to “An account was successfully logged on” can be seen during this time.
Attacks on the Apache Server began at 5 PM and ended 9 PM on 3/25/20. During this time, a successful “Brute Force” Attack occurred on VSI’s computer network. The primary location of the attack was from the Ukraine. The attack is notable mainly because it was preventable. Or at least could have mitigated while it was occurring with the proper Network and Administrative controls in place.
- To mitigate this issue in future I recommend the VSI start working toward a DiD strategy by implementing the following solutions:
- Administrative
- Strong passwords using special characters, minimum length, and set to expire every 90 days.
- Account lockout policy set to 5 failed logon attempts and disabled for at least 1 hour.
- 2 Factor Authentication.
- Network
- Block or limit access to “VSI_Account_logon.php” from non-US IP addresses if possible.
- Review “Deleted Accounts” Alert and find out why it did not trigger.
- Create new Alert for “Account was locked out.” And run it every 15 minutes. With a threshold of 13.
- Add a panel to the Windows Server dashboard that monitor’s Windows events by status.
- Deny access to known rouge UA’s.
- Install a Web Application Firewall.
- Use IDS/IPS like Snort.
- Install a Nessus scanner.
- CATPCHA on Website forms.
