A Journey Through Adversary Infra Analysis- Learning to Pivot

 Recently Fortinet released an advisory stating two CVEs (CVE-2024–55591 and CVE-2022–40684) were actively exploited in the wild. I started looking from a Threat hunt perspective but as there are few details in the advisory to map out the overall picture — I began to turn to adversary infra analysis in hopes of finding connections to expand observables and patterns. From IOCs from a Fortinet advisory, it expanded to correlation with Arctic Wolf’s “Console Chaos” research and other pivots with potential connections to the Core Werewolf (also known as PseudoGamaredon or Awaken Likho) threat actor group.

Fortinet Advisory

I spilled my coffee a bit when I first looked at the Fortinet advisory mentioning the usual suspects — loopback addresses and Google DNS. But then re-reading the advisory and these in context gave me some perspective and understood why it was mentioned. While acknowledging the context in which these can be important but ignoring the “values”, I focused on the other IOCs in the report.

Thanks for reading! Subscribe for free to receive new posts and support my work.

Understanding Infra Analysis

Infra-analysis is not entirely part of threat hunting but it aids in threat hunt research. So the scope of it is different from let’s say a threat intel OR cybercrime analyst. This analysis helps us to examine how systems, services, and networks that adversaries may use to conduct their operations and help us understand:

Tools and techniques adversaries use within a cluster or campaign

• Operational patterns and preferences
• Potential connections to known threat actors/group’s clusters/campaigns

In this article, I will focus on several “breadcrumbs” of infra analysis we can follow:

1. Service Fingerprinting: Examining unique identifiers of services running on different ports

2. Historical DNS Resolution: Understanding domain name patterns and relationships

3. SSL Certificate Analysis: Identifying relationships through shared certificates

4. Network Patterns: Looking for patterns in how services are deployed

5. ASN Patterns: Understanding the adversary’s preferred hosting providers and regions

Note: Adversary infrastructure tracking is complex with several layers within the group and between groups. So, the connections we make solely based on network observables may not be accurate to “make” connections. That being said, it is still valuable to understand suspicious ASNs, “commonalities” between groups, tools we uncover, and several other reasons.

All this to say, take this with a grain of salt in terms of “connections” but enjoy the things we uncover and learn along the way.

Pivoting structure

I would like to have some structure so we can expand our pivots to multiple points while still validating them. Here is my approach from Threat Hunt perspective.

1. First-Order Analysis: “Staying close to the source”

• I prefer direct IOCs from the primary source that kicked off the analysis (Fortinet advisory)
• Identify ports and services and pivot from there
• Check SSL certificates to find past and future patterns

2. Correlation: For any additional references we find for the above, we should

• Check the findings and timeframes of the research
• validate TTPs if available

3. Expand Infrastructure:

• Use the patterns to see if we can find similar infrastructure (not all here can have high confidence unless there are solid connections)
• Check historical DNS, certificate, and ASN data from the previous two steps

We are talking about the same thing: Arctic Wolf

when searching for IOCs from Fortinet’s advisory, I came across Arctic Wolf’s “Console Chaos” blog and saw three IP addresses matched. One thing I always like to check is if they are connected to each other by analyzing TTPs — which matched too in this case. So, we can connect words we initially observed like “exploited in the wild” to actual findings from the intrusion set. This is not technically a pivot but it is always good to correlate our dataset with multiple sources.

My failures: VPS Scanning Infra

I started digging into the overlapping IPs but it did not reveal much as most of these IPs were on VPS providers like DigitalOcean and Datacamp, and from the “Console Chaos” blog, these are connecting to the HTTP interface for likely confirmation purposes.

1. 45.55.158.47 has below both do not provide any more pivots just from these fingerprints:

45.55.158.47 (DigitalOcean):

- Primary Purpose: Reconnaissance

- Service Configuration:

- Port 22: SSH (Fingerprint: cb889df05d1536cc39938ac582e1eca1090e3662a270abff55ca6b119f2f11e0)

- Port 3306: MySQL (Fingerprint: e77fd0556cd803fa2613509423aaa86d34dda37e19cc2cfce7145cd9b1f5ff6a)

It’s the same for 37.19.196.65 and 155.133.4.175 and a few others. We like to see results in infrastructure analysis from all pivots but most of the time we just don’t or we may find something irrelevant.

2. 87.249.138.47 has:

- appears to be from compromised synology (quickconnect.to)

3. 167.71.245.10 has:

- Port 22: SSH (Fingerprint: b0ac2b8a274766825b0588520b9154995c3301cd8ae9a0020621af1529ea9179530)

- Port 3389: RDP

- Historical: emojis.soon.it (October 2024)

Possible C2: Something Interesting but for Future

Moving on the list, 157.245.3.251 has some interesting pivots although did not produce solid results.

- Operating System: Ubuntu Linux

- Service Layout:

- Multiple SSH services (both standard and non-standard ports):

- Port 22 (Standard)

- Port 222 (Alternative)

- HTTP services via Nginx on high ports (30000–30005)

- Historical ports (2024–09–09 and 2024–09–10): 9527, and 9530 returned empty responses but the timeframe was still in our research range.

We can craft a query to find everything on those ports and return empty responses but I did not find any. Regardless, it is an interesting pivot we can query in the future, in case it returns back.

(((services.port:{22,30002,30003,30004,30005})

AND (services:HTTP and services:SSH))

and autonomous_system.name: "DIGITALOCEAN-ASN"

and services.banner_hashes: "sha256:97f8ff592fe9b10d28423d97d4bc144383fcc421de26b9c14d08b89f0a119544"

and services.tls.certificates.leaf_data.issuer_dn: "C=US, O=Let's Encrypt, CN=R11")

and services.software.vendor='Ubuntu'

This query is to identify infrastructure with a similar pattern:

- Multiple HTTP services on specific high ports

- SSH presence

- Specific SSL certificate patterns

- Ubuntu-based systems

USPS Phishing Infra

When looking at 23.27.140.65, I found what looked like a USPS phishing lure from historical domains. This can be of interest here as it could be either for phishing for initial access with a focus on package tracking themes from the group itself or usage of possible Initial Access Broker (IAB) for that purpose. It’s hard to tell just from network observables but nonetheless, we have 102 domains used in the past 9 months.

- Timeline: July-December 2024

- Target Focus: USPS-related services

- usps.packages-usa[.]com

- usps-do-track[.]com

- usps.packages-ui[.]com

- and many more

The Core Werewolf Connection:

My most interesting discovery so far related to a previous campaign was the investigation of 31.192.107.165 (ORG-LVA15-AS). According to bi-zone, this network observable was previously linked to Core Werewolf (around October 2024), which matched our timeframe, too.

- Domain: conversesuisse.net

- Server — Werkzeug/2.2.2 Python/3.11.8

- MySQL 8.0.35–0ubuntu0.22.04.1

- Redis on 6379

- SSH on 6579

I spent some time looking for patterns mentioned in the blog in the same ASN and found about 49 IP addresses running Telegram-related content (one of the C2 mentioned).

Some other things that stood out are related to Silver C2 and Bulletproof services.

- 141.105.64.204

- services.ssh.server_host_key.fingerprint_sha256: f8431322831c2508255655d6d4b14802e68be656cec9189f109eb386032e99be

- 3524 results — not all possibly related to the infrastructure but a few interesting ones I found related to silver C2 and about 2,216 are from bulletproof services.

Staying within our bounds

149.22.94.37 revealed connections to an executable on VT

• PDF-themed executables probably via SEO Poisoning (PDFs Importantes.exe)
• Persistence via Registry modification for startup:

%windir%\System32\reg.exe add "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run" /v "KeybordDriver" /t REG_SZ /d "\"%APPDATA%\Windows Objects\wmimic.exe\" winstart" /f

• Malware dropped as likely second stage
• - C:\WINDOWS\system32\wmihostwin.exe
• - C:\WINDOWS\system32\wmiintegrator.exe
• - C:\WINDOWS\system32\wmimic.exe
• - C:\WINDOWS\system32\wmisecure.exe
• - C:\WINDOWS\system32\wmisecure64.exe
• Network enumeration with net view commands

What does this mean? Well, nothing, because if we check the creation date 2017–03–26 05:14:46 UTC and submission date 2020–09–25 15:34:52 UTC, it is far off from our research bounds and might be linked to some other cluster/campaign. Can there be a connection? Probably. Can we make that connection now? Well, no, because of the timeframe and no other evidence to support it.

Infrastructure Relationship thus far

The investigation revealed several distinct but possibly (some are very distant) interconnected infrastructure components:

1. Scanning: for target identification and reconnaissance

2. Phishing: Likely for credential harvesting and initial access

3. C2: Things we can look to connect in the future or at least have them in our dataset to extract more insights like C2 configs, ports, etc

4. Campaign: Core Werewolf (AKA PseudoGamaredon, Awaken Likho) threat actor group relationship during the same timeframe. It requires additional patterns and observables to strengthen our intuition.

It’s the timing

In the middle of all this, I came across a social media post from “Belsen_group” claiming to have credentials and configs. They published this as “free” and organized it via country “to make it easier for you”. I checked their site and the data to get an overall sense to see if it is related to current exploitation. I will not go a lot into it as Kevin Beaumont beat me to it. According to Kevin, these are from 2022 for CVE-2022–40684 exploitation and not this one. It’s just the timing and you can find the article at https://doublepulsar.com/2022-zero-day-was-used-to-raid-fortigate-firewall-configs-somebody-just-released-them-a7a74e0b0c7f

Several passwords I observed, one might say, are weak in nature. They are mostly have patterns of

1. Simple 4–8 char

2. Passwords are a variation of the username of variation of VPN

The IPs as part of the exposure are provided by Amram Englander on their GitHub https://raw.githubusercontent.com/arsolutioner/fortigate-belsen-leak/refs/heads/main/affected_ips.txt, so feel free to check them and change config accordingly.

Why bother then

If nothing is related, why publish this? My only intention was to provide how infra analysis can help threat hunters at times and show what I’ve learned:

1. Patience — a lot of dead ends, so, stay focused

2. Finding pivots — No pivot is a sure shot and staying in bounds during our pivots. We should always understand why infra is structured in certain ways and how it can be used.

3. Using multiple data sources — certificates, DNS records, and service fingerprints from multiple platforms is always a good idea.

For anyone starting out in infrastructure analysis, remember to document everything (especially the failures) to help in the future and stay excited.

References

1. https://arcticwolf.com/resources/blog/console-chaos-targets-fortinet-fortigate-firewalls/

2. https://bi-zone.medium.com/core-werewolf-hones-its-arsenal-against-russias-government-organizations-7fbe8cc58b27

3. https://www.facct.ru/blog/core-werewolf/

4. https://securelist.com/awaken-likho-apt-new-implant-campaign/114101/

Thanks for reading! Subscribe for free to receive new posts and support my work.

The Methodology of Threat Hunting Research

 

Recently, after discussions with folks, I realized there may exist few challenges in threat hunting, especially related to threat hunt research. These folks understand that threat hunting is a proactive approach to uncovering threats, but they feel their approach is scattered and inefficient, leading to inconsistent results despite spending much time. They felt every hunt was like starting from scratch and were unable to build from their previous hunts or scale their process effectively. I think one of the possible root causes might be the lack of a structured approach to threat hunting research. Without this structure threat hunting will become a game of chance rather than a systematic process.

But threat hunting research can be an efficient and repeatable process to deliver consistent results. To achieve that we need to understand and implement a structured methodology, especially for the research phase that can navigate us in all other phases of threat hunting process.

Threat Hunt research is all about figuring out how adversaries operate and approaches to uncover their presence but it is also the foundation to conduct effective hunts. It's a process of understanding adversary behaviors, mapping them to observable events, exploring hunt hypotheses, and understanding environmental factors with structured methodologies, frameworks, and approaches. Hunt research is the foundation that helps us (threat hunting teams) go from an ad-hoc activity into a repeatable, scientific process.

I will attempt to compare threat hunting research to DFIR (Digital Forensics and Incident Response) drawing from my experiences whenever possible.

TLDR: Here is the diagram that might help you in threat hunting research

Threat hunting research is similar to Digital Forensics and Incident Response (DFIR), in the latter folks look into incidents after they happen and the former find patterns before potential breaches or try to identify patterns deviating from the environmental norms. But the way we investigate is similar. Knowing these similarities can help us better approach threat-hunting research.

1. Initial Research

Just like DFIR investigators plan their investigations before they start, the hunting research phase starts by planning and thinking about what they might find (consider the A.P.E.X. framework to see how). In DFIR, investigators might ask, What systems were probably compromised? and What clues would show that they were?

Similarly, in threat hunting research we should ask,

• What specific attacker behavior am I investigating?

• What does the attack chain look like?

• What technical components make up this behavior?

• What does a successful execution of this behavior look like?

The planning phase is important for both because it helps us understand the overall picture. DFIR investigators create timelines of events and figure out the scope/pivots, while threat hunting research makes maps of the past attack patterns of an adversary and figures out the possible patterns of the adversary behavior in the future to formulate possible hypotheses, hunt pipeline to better assist/enhace existing detections.

2. Data Source Analysis:

DFIR investigators find and record important evidence when they investigate an incident. They determine which sources, artifacts, and means might contain evidence of the incident. Threat hunting research do something similar when they examine data, but with a proactive twist. Teams sometimes spend too much time to figure out which data sources might contain relevant information at the time of the hunt. Instead, research should include a systematic approach to map adversary behaviors or patterns with relevant, available data sources and fields before even beginning the hunt. It will help us understand not just the data sources we need but the categories in those data sources along with their fields. We need to know:

• Where the relevant data is in the environment

• How the data is formatted and structured

• How long it’s supposed to be kept and how it’s collected

• How different data sources are related

3. From Artifacts to Activities:

While IR investigators piece together clues from various sources to figure out what happened during an incident, threat hunt research does the same thing, but in reverse. We try to guess what artifacts and patterns would be left behind by specific adversaries/techniques/tactics based on our hypothesis. The main difference is that DFIR looks back to figure out what happened, while we try to look forward to predicting future attacks. However, the intelligence gathered by both can help us understand attack patterns and use them to detect future attacks.

The key difference between hunters who feel like random searching and the ones who do structured hunting is hypothesis generation.

A good hypothesis in threat hunting research: (Check out the Hunting Opportunities section here)

• Focuses on a specific attacker behavior or technique

• Links to concrete, observable events in the environment

• Accounts for variations in how the behavior might manifest

• Considers the context of environment and normal operations

• Includes clear criteria for validating or disproving the hypothesis

4. Environmental Analysis

DFIR teams need to know the environment they’re investigating which includes the network architecture, security controls, and business context (which might not apply to consulting or MSSP). In an effective threat hunting research, we know that no two environments are identical and what works in one organization might not work in another due to differences in:

• Available data sources and logging configurations

• Infrastructure complexity and distribution

• The security tools that are deployed

• The typical patterns of business activity

• The limitations and blind spots in logging and monitoring

This understanding helps DFIR teams spot unusual activity and helps threat hunting research distinguish between normal and potentially malicious/suspicious activities. Knowing these environmental factors also helps us to develop both effective and efficient hunts.

Thanks for reading! Subscribe for free to receive new posts and support my work.

5. Documentation

DFIR investigations require extensive documentation – from initial findings through final conclusions. This documentation becomes a treasure trove for future investigations, helping teams spot patterns across multiple incidents. Threat hunting research is also not a one-time effort - it's an evolving process that builds upon itself which in turn refines methodology rather than specific incidents. Each hunt provides new insights, reveals gaps in visibility, and helps refine future hunting approaches. Maintaining detailed documentation of our research, including both successes and failures, is crucial for building a mature hunting program. Both types of documentation have some things in common:

• Detailed technical findings and observations

• Analyzing patterns and behaviors

• Figuring out how different pieces of evidence are connected

• Recommendations for future investigation or hunting

Research —> Hunt

The reason we did all the research is to hunt in our environment. It is also a catch-22 - Do we figure out what to research first and then hunt? or Do we figure out what to hunt and then kick off research? This transition from theoretical research to practical hunting is often time-consuming, and resource-intensive. It also requires several things to fall in place to bridge the gap between knowledge and action.

• Clear documentation of hunting methodology - not observable

• Specific queries or search parameters from the 2, 3, 4, and 5 steps

• Success criteria and metrics

• Plans for handling findings - from clear security incidents to suspicious but irrelative (to our hypothesis) findings to new hunts

Validation

DFIR teams and threat hunting researchers both know that false conclusions can be expensive. They both validate their findings through multiple data sources and analysis techniques - sometimes it requires simulating the behavior in a test environment. Threat hunting research requires thorough validation of hypotheses and methodologies.

• Testing assumptions, queries, and hunt methods against known-good data or "normal behavior"

• Verifying findings through multiple sources along with observable events actually indicate the behavior we are hunting for

• Considering alternative explanations

• Documenting limitations and caveats

Adaptability

Just as DFIR teams must adapt their techniques/methodologies to new types of attacks, evolving tech, and complex environments, threat hunting research needs to keep getting better. Both face similar challenges:

• Changes in attacker techniques and tools

• Evolution of enterprise technology and architecture

• changes in security controls and monitoring

Building flexibility into our research methodology allows us to:

• evaluate new sources of evidence and data

• Scale across growing environments

• Integrate new tools and technologies or refine the ones according to our requirements.

Conclusion

Hopefully, this parallel between DFIR and threat hunting provided valuable insights into threat hunting research. Both require systematic approaches, detailed technical understanding, and the ability to adapt to changing circumstances. By applying a similar standard methodology of DFIR to threat hunting research, organizations can make their hunting programs even better. There is always room for creativity and innovation in this structured approach and methodologies as threat hunting is both an art and a science.

A.P.E.X: Threat Hunting Through Structured Hypothesis Generation

 

In a laest report I worked on Hunters International ransomware, I provided several hypotheses you can implement in your environment. I want to reiterate the importance of integrating environmental context with threat actor profiling, a crucial aspect of threat hunting. I want to re-introduce the A.P.E.X. framework, which we used in our other threat reports, and delve a little deeper into its implementation with an example. A.P.E.X. hypothesis generation framework is a comprehensive methodology I developed internally for hypothesis generation. It is

Four-Element Structure: The A.P.E.X. framework integrates both perspectives through four key elements: Analyzing the environment, Profiling threats, Exploring anomalies, and examining X-factors (unexpected variations). This integration proved crucial in our investigation. The framework offers a structured, and adaptable methodology that aids threat hunters in generating hypotheses for both known and emerging trends.

• X-Factor Consideration: A.P.E.X. explicitly accounts for unexpected or unexpected threat techniques through its X-factor element. This approach hopes to help hunters account for emerging threats.
• Hypothesis Generation: A.P.E.X. offers a structured approach to creating both primary and focused hypotheses, providing more granularity and direction for threat hunters.
• Expected Observations: For every hypothesis, A.P.E.X. outlines detailed expected observations, categorized into standard (E) and unexpected (X) observations. This can also be translated into success criteria.
• Integration of Multiple Techniques: The framework takes various threat-hunting "techniques", including hypothesis-driven hunting, anomaly detection, and threat intelligence integration.

Theory to Practice:

A.P.E.X. can be used to generate the following hypothesis:

"Attackers are exploiting Oracle WebLogic debug ports for initial access."

A (Analyzing the environment): Oracle WebLogic servers were present in the environment, and debug ports were potentially exposed.

P (Profiling threats): Hunters International has shown a pattern of exploiting web server vulnerabilities.

E (Exploring anomalies): The unusual connections to port 8453 (WebLogic debug port) were noticed in the network logs.

X (X-factor): Attackers might use novel exploitation techniques or zero-day vulnerabilities.

Expected Observations:

E1: Connections to port 8453 from external IP addresses.

E2: Execution of Java processes with debug parameters.

E3: Creation of web shells or other backdoors post-exploitation.

X1: Unusual Java class loading or reflection activities indicating a new exploitation method.

X-factor consideration leads to unique insights:

We explicitly included the X-factor in the overall hypothesis generation process to account for novel or unexpected attacker behaviors based on patern-projections. This approach allows threat hunters to project and hunt emerging threats that may not fit already established patterns observed. In this case, it helped us discover an exploitation technique for Oracle WebLogic servers. Considering beyond known tactics, techniques, and procedures (TTPs), the X-factor enables a more forward-looking threat-hunting approach. While monitoring for expected observations (E1-E3), we also looked for X1 - unusual Java activities. This led to the finding of a previously unknown method of exploiting WebLogic.

• Attackers used a custom Java class to execute arbitrary code.
• We observed unusual Java class loading patterns that didn't match known exploit techniques for the group.

This approach generated specific, actionable observations and helped track the other activities in the attack lifecycle:

E1: Connections to port 8453 helped identify potential initial compromise, scoping, and timeline.

E2: Execution of Java processes with debug parameters revealed the attacker's method of maintaining access.

E3: The creation of the "Intel" folder and China Chopper web shell was detected allowed us to identify compromised systems quickly.

X1: Unusual Java class loading was observed, indicating a potential new exploit technique.

By following these detailed observations, I was able to track the attack progression from initial access through lateral movement and ultimately to the ransomware deployment, providing a comprehensive view of the Hunters International operation and enabling more effective intrusion analysis and hunting strategies.

Hope this helps. Happy Hunting!!

Threat Hunting: Decoding User-Agents for better Insights

User-agent analysis is one of the powerful methods in threat hunting to identify unusual behaviors that may indicate malicious activity. Each connection tells a story - from the mundane to the suspicious.

In this post, I'll share how we can use user-agent analysis to spot the needles in the haystack. We'll explore from basic statistical techniques to a real-world case where an unusual user-agent pattern helped me find unauthorized remote access software unusual to our environment. While we can hunt for attackers who try to hide from fancy detection tools, user-agent analysis can also reveal activities hiding in plain sight.

TL;DR:

• What: An attempt to find suspicious activity through user-agent analysis in proxy/network logs

• Why: User-agents can help us identify unauthorized tools and suspicious behavior

• How: Using statistical analysis (Z-scores) and a custom suspicion scoring to identify anomalies

• Win: found unauthorized remote access software (RemotePC) used by a compromised host via WebSocket user-agent pattern

• Takeaways: Includes SQL queries, user agent investigation decision tree, and common false positive examples

Why it matters to us:

1. Finding Anomalies: Rare or uncommon user-agents like those associated with remote management (RMMs) or automation tools and might help identify unauthorized or compromised users/hosts.

2. Identifying Pattern: Statistical analysis of user-agents helps detect deviations from the normal in our environment. Metrics like suspicion scores (which we’ll discuss in more detail later), unique hosts, total requests, and z-scores quantify anomalies. For example, an unauthorized remote management tool’s user-agent with an low or in-bursts "active_days_z" score and only observed from single user/host could indicate potentially suspicious activity.

3. Behavioral indicators: Combining user-agent data with metrics like total requests and unique hosts provides a comprehensive view of user behavior. A high "unique_hosts" count with a remote access user-agent might suggest “normal” usage in our environment. In contrast, a low "unique_hosts" count could indicate a one-off occurrence or “unauthorized” in the environment.

Let’s play with some Stats:

Gathering data:

We are collecting proxy or network data with user agent info in our SIEM. This data typically includes username, IP, protocol, category, HTTP method, duration, bytes in and out, and other relevant details. Most proxy services like Forcepoint Websense and Imperva provide this data. If a data source only has user agent and IP information, we can cross-correlate it with other sources to obtain user and host details. For simplicity, we can also use the IP address as a means to count as an "entity".

The initial query to gather user agents would be something like:

SELECT 
    dst_host,
    user_name,
    event_time,
    duration,
    http_method,
    protocol,
    ua
  FROM logs, UNNEST(user_agent) AS ua // because for me user_agent is an array field. remove it if its not in your dataset

To better organize and process data, we use Common Table Expressions (CTEs) in SQL. We then use the info from the above CTE to create user agent-specific metrics likeunique hosts, usernames, along with averages and standard deviations.

user_agent_stats AS (
  SELECT 
    ua,
    COUNT(DISTINCT dst_host) AS unique_hosts,
    COUNT(DISTINCT user_name) AS unique_users,
    COUNT(*) AS total_requests,
    COUNT(DISTINCT DATE(event_time)) AS active_days,
    AVG(duration) AS avg_duration,
    STDDEV(duration) AS stddev_duration,
    COUNT(DISTINCT http_method) AS num_http_methods,
    COUNT(DISTINCT protocol) AS num_protocols,
    MIN(event_time) AS first_seen,
    MAX(event_time) AS last_seen
  FROM unnested_logs
  GROUP BY ua

Next, we calculate the Z-scores which measure how unusual each user-agent's behavior is compared to the overall environment. A Z-score tells us how many standard deviations a particular value is from the average.

 Z-score = (observation - mean) / standard deviation
 Typical thresholds:
		- |Z| > 2: Unusual
		- |Z| > 3: Very unusual
		- |Z| > 4: Extremely unusual

In our set, each metric (hosts, users, requests, active days, duration), we:

• Subtract the global average (e.g., ua.unique_hosts - gs.avg_unique_hosts)

• Divide by the standard deviation ( gs.stddev_unique_hosts)

• Use NULLIF() to prevent division by zero errors

(ua.unique_hosts - gs.avg_unique_hosts) / NULLIF(gs.stddev_unique_hosts, 0) AS unique_hosts_z,
(ua.unique_users - gs.avg_unique_users) / NULLIF(gs.stddev_unique_users, 0) AS unique_users_z,
(ua.total_requests - gs.avg_total_requests) / NULLIF(gs.stddev_total_requests, 0) AS total_requests_z,
(ua.active_days - gs.avg_active_days) / NULLIF(gs.stddev_active_days, 0) AS active_days_z,
(ua.avg_duration - gs.global_avg_duration) / NULLIF(gs.stddev_avg_duration, 0) AS avg_duration_z,
LENGTH([ua.ua](<http://ua.ua/>)) AS ua_length
(ua.total_requests / NULLIF(ua.unique_users, 0)) AS requests_per_user,
    (ua.unique_hosts / NULLIF(ua.unique_users, 0)) AS hosts_per_user

Understanding suspicion_score

In our example, we created a ‘suspicion_score’ to analyze user-agent strings and identify potentially anomalous activity (this is just a sample one and you can interpret and customize it based on your environment and expertise). It calculates a suspicion score (0 to 1) for potentially anomalous user agents by weighing multiple “attributes”. For example, we gave 15% weight to unusual numbers of hosts, users, requests, and suspicious user agent strings, 10% weight to activity patterns, and 5% weight to consider protocol diversity and user agent categories. These weights are subjective and can be experimented based on the environmental knowledge or goal. This overall score also helps us prioritize our investigations and focus on the most anomalous behaviors first.

(
0.15 * (1 - 1 / (1 + EXP(-ABS(unique_hosts_z) / 10))) +
0.15 * (1 - 1 / (1 + EXP(-ABS(unique_users_z) / 10))) +
0.15 * (1 - 1 / (1 + EXP(-ABS(total_requests_z) / 10))) +
0.10 * (1 - 1 / (1 + EXP(-ABS(active_days_z) / 5))) +
0.10 * (1 - 1 / (1 + EXP(-ABS(avg_duration_z) / 5))) +
0.10 * LEAST(num_http_methods / 10, 1) +
0.05 * LEAST(num_protocols / 5, 1) +
0.15 * CASE WHEN ua_length < 20 OR ua_length > 500 THEN 1 ELSE 0 END +
0.05 * CASE WHEN ua_category = 'other' THEN 1 ELSE 0 END
) AS suspicion_score

suspicion_score variable combines different attributes:

• Statistical anomalies: We use Z-scores for unique hosts, users, total requests, active days, and average duration. These help identify outliers in our environment.

• HTTP methods and protocols: We consider the variety of methods and protocols used by the user_agent.

• User-agent characteristics: We examine the length of the user-agent string and its category, flagging unusually short or long strings and those categorized as "other." However, this approach isn't very effective and can overlook things where a perpetrator or insider uses a slight variation of a "normal" user-agent. But it is good for our example

Why we did what we did?

https://www.datawrapper.de/_/dmTsd/?v=2

An example: Unusual WebSocket Activity

As threat hunting starts with a hypothesis, Mine is based on APEX Framework (A - Analyze Environment, P - Profile Threats, E - Explore Anomalies, X - X-factor Consideration)

[A] Based on our environment's baseline of approved remote management tools and their typical patterns, [P] we hypothesize unauthorized remote access tools are being used for malicious purposes. [E] We expect to find anomalous user agent strings with unusual activity patterns like:

• Rare or unique user agents on few or single hosts/users

• Inconsistent version patterns across the environment

• RMM tool usage

[X] Additional factors include correlation with suspicious domains, unexpected protocol usage, and deviations from known tool signatures.

I noticed a unique UA (User Agent) which was interesting, websocket-sharp/1.0/net45 due to its behavior and sudden spikes in activity from the metrics.

active_days: 14

active_days_z : 6.065310525407249

avg_duration :6.457142857142856

avg_duration_z: -0.5618276785969151

first_seen:2024-10-01T23:11:21Z

hosts_per_user:1

last_seen:2024-10-29T20:17:13Z

num_http_methods: 1

num_protocols: 1

requests_per_user: 35

suspicion_score: 0.514988557364753

total_requests: 35

total_requests_z: -0.00699213278186984

ua_category: other

ua_length: 19

unique_hosts: 1

unique_hosts_z: -0.008524453434536202

unique_users: 1

unique_users_z: -0.016810278172976426

user_agent: websocket-sharp/1.0

This WebSocket client library seems interesting to me because it's not typical in our environment and it only appeared for one user for a short activity period.

Analysis:

1. Finding anomaly: websocket-sharp/1.0/net45 in our traffic logs suggests either a custom application using it or something to investigate

2. Gathering statistics:

   • Active for 14 days (UA was seen on 2024-10-01T23:11:21Z but actively making requests for 14 days), with an active_days_z score of 6.065,

   • 35 total requests from a single user and host

   • Utilized only one HTTP method and one protocol

   • Suspicion score of 0.515, which is an anomalous behavior in our environment

All in all, we can see a link between a single source and some patterns that “deviate” from “normal”. The elevated suspicion score and abnormal active days z-score in our environment helped us to pick this to investigate first out of all the other things in our dataset

1. Connections: Investigating into unusual WebSocket user agent took an unexpected turn. At first, I thought they might be suspicious, but analysis revealed two really interesting connections.

   1. I found this WebSocket connection that looked a bit suspicious wss://capi.grammarly.com/fpws?accesstoken. Here I was getting excited to find a suspicious WebSocket connection, and it turned out to be someone's grammar checker. it reiterated that not every unusual connection is malicious..

   2. This appears to be a common thing among multiple vendors like mdds-i.fidelity.com, uswest2.calabriocloud.com/api/websocket/sdc/agent as HTTP/s connections.

2. I did some more digging and found some interesting connections. I found connections to wss://est-broker39.remotepc.com/proxy/remotehosts?, a domain associated with RemotePC, a remote access service, with single username on a single host.

What we found:

• User was infected via a masqueraded name of a RMM tool downloader (additional analysis confirmed this)

• Unauthorized use of a legitimate service

• Here are some other findings from the same dataset using the same method:

  • Several outdated versions of multiple RMM tools that are legit in the environment

  • Usage of other RMM tools that are not approved

Additional Improvements to Experiment With

1. User-Agent Length

   • Try dynamic thresholds based on historical data

   • Create UA length profiles per application category (browsers vs. API clients vs. mobile apps)

   • Track sudden changes in UA length patterns over time

2. More stasts

   • Try percentile-based:

   • Try Median Absolute Deviation (MAD) to find better outliers

3. User-Agent Categorization

   • Try different categorization

   • Track categories over time

   • Create allow/deny lists per category based on environmental context

   • See if you can implement behavioral profiling per category

4. Suspicious Score

   • Play with suspicious scores (Z-score-based, Volume-based, Behavioral based) and weights

   • The current implementation will not work if duration or any other parameters are not present, so, we can experiment to calculate it even any of the parameters are null.

   • Volume-based patterns:

     • Sudden spikes

     • Regular patterns that could indicate automation

   • Behavioral indicators:

     • Add enrichments with geo-location for domain, IP , ANS etc

     • Protocol usage

Common User-Agent Success and False Positive

Here’s a lesson I learned the hard way in threat hunting: defining your success criteria upfront. Before you begin any hunt, make sure to establish clear criteria for what specifically confirms a threat. Second, what might look suspicious but turn out to be normal business activity? This saves time, helps document/cross-check legitimate business activities, and helps focus the investigation on leads. These are based on our hypothesis, anything else you find should be in our threat hunt pipeline and/or marked for future investigations.

Success Criteria:

• Unauthorized RMM tool confirmed

• Abnormal usage patterns documented

• Host/user attribution established

• Tool installation timeline determined

False Positive Criteria:

• Legitimate business tool

• Approved user/purpose

• Normal usage pattern

• Documentation exists

Example of False Positives or need to be documented:

 https://www.datawrapper.de/_/W6weg/

User Agent Analysis Decision Tree

When we spot suspicious user-agents, a decision tree helps us figure out where to start investigating. Start with any user-agent that caught our attention (either because it has a high suspicion score or because it looks weird). Follow the path based on what you find, and make sure to write down everything you do at each step. For instance, when we saw “websocket-sharp/1.0/net45”, we first checked if it was already on our list (it wasn’t). Then, we looked at it against known software (it was a RemotePC connection), checked how often it was used (it was only on one host and had a lot of activity), and then moved on to “Further Investigation”. That’s when we found out that someone was using unauthorized RMM software. Remember, this tree is just a way for me to organize my thoughts, not a strict rulebook. Over time your environment and goals will guide you on how to approach each decision point.

These findings highlight the value of user agent analysis in threat hunting. The discovery of both benign (Grammarly) and potentially concerning (RemotePC) connections tell us the necessity of understanding that anomalies are just that—anomalies. Manual investigation, further research, and correlation with other data sources are required to figure out why an anomaly exists and what it means in our environment. This helps us find more interesting results and data for further analysis. Hopefully, it was helpful how user-agent analysis in combination with statistical anomalies can help each other to uncover hidden activity, linking unusual patterns and some ideas on how we can approach user-agent analysis.