Command execution vulnerabilities account for a significant proportion of disclosed Java vulnerabilities. This chapter analyzes the general principles of command execution, selection of hook points, real-world vulnerability cases, and detection algorithms.

1. Java Command Execution APIs

In Java, command execution can be achieved through the following APIs:

• java.lang.Runtime.exec()
• java.lang.ProcessBuilder.start()
• java.lang.ProcessImpl.start()

The most commonly used API is Runtime.getRuntime().exec(), used as follows:

Runtime.getRuntime().exec("touch /tmp/1.txt");

The Runtime.exec() method actually has six overloaded variants:

public Process exec(String command)
public Process exec(String command, String[] envp)
public Process exec(String command, String[] envp, File dir)
public Process exec(String cmdarray[])
public Process exec(String[] cmdarray, String[] envp)
public Process exec(String[] cmdarray, String[] envp, File dir)

The first five methods all eventually delegate to the last one. Therefore, we focus on the third and sixth overloads.

public Process exec(String command, String[] envp, File dir)
    throws IOException {
    if (command.length() == 0)
    throw new IllegalArgumentException("Empty command");
    StringTokenizer st = new StringTokenizer(command);
    String[] cmdarray = new String[st.countTokens()];
    for (int i = 0; st.hasMoreTokens(); i++)
    cmdarray[i] = st.nextToken();
    return exec(cmdarray, envp, dir);
}

All exec() overloads ultimately invoke the following method:

public Process exec(String[] cmdarray, String[] envp, File dir)
    throws IOException {
    return new ProcessBuilder(cmdarray)
        .environment(envp)
        .directory(dir)
        .start();
}

As shown in the source code, Runtime.exec internally constructs a ProcessBuilder instance and calls its start method.

 2. Underlying Call Chain

The two most common command execution APIs are java.lang.Runtime.exec() and java.lang.ProcessBuilder.start(). More low-level entry points also exist, such as java.lang.ProcessImpl.start(). Below is a demonstration of various command execution techniques:

import java.lang.reflect.Method;
import java.util.Map;

public class Main {
    public static void main(String[] args) throws Exception {

        // Define commands
        String command = "touch /tmp/1.txt /tmp/2.txt /tmp/3.txt";
        String[] commandarray = {"touch", "/tmp/1.txt", "/tmp/2.txt", "/tmp/3.txt"};

        // Method 1: Runtime.exec()
        Runtime.getRuntime().exec(command);

        // Method 2: ProcessBuilder
        new ProcessBuilder(commandarray).start();

        // Method 3: Direct invocation via reflection on ProcessImpl
        Class clazz = Class.forName("java.lang.ProcessImpl");
        Method method = clazz.getDeclaredMethod("start", new String[]{}.getClass(), Map.class, String.class,  ProcessBuilder.Redirect[].class, boolean.class);
        method.setAccessible(true);
        method.invoke(null, commandarray, null, ".", null, true);
    }
}

Tracing through the JDK source reveals that all command execution paths ultimately converge at the native method java.lang.UNIXProcess.forkAndExec. Its declaration is as follows:

// from: jdk11/src/java.base/unix/classes/java/lang/ProcessImpl.java
private native int forkAndExec(int mode, byte[] helperpath,
                               byte[] prog,
                               byte[] argBlock, int argc,
                               byte[] envBlock, int envc,
                               byte[] dir,
                               int[] fds,
                               boolean redirectErrorStream)
    throws IOException;

This is a native method implemented in C/C++. To observe the call stack during execution, we can trigger an exception inside forkAndExec() while debugging (see Figure 1-1).

The resulting stack trace is:

Exception in thread "main" java.lang.SecurityException: rce block by rasp!
     at java.lang.UNIXProcess.forkAndExec(Native Method)
     at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
     at java.lang.ProcessImpl.start(ProcessImpl.java:134)
     at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
     at java.lang.Runtime.exec(Runtime.java:621)
     at java.lang.Runtime.exec(Runtime.java:451)
     at java.lang.Runtime.exec(Runtime.java:348)
     at Main.main(Main.java:12)

This test was run on a Unix-like system, where the final implementation class is java.lang.UNIXProcess. On Windows, the call stack differs slightly. Figure 1-2 summarizes the command execution call flow across operating systems.

3. Hook Point Selection

Traditional RASP solutions typically hook at the first layer shown in Figure 1-2—specifically, the <init> or start methods of java.lang.ProcessImpl (JDK 9+) or java.lang.UNIXProcess (JDK 8 and earlier). These are pure Java methods and can be instrumented directly via bytecode manipulation.

However, since the actual OS-level command execution occurs in the native method (forkAndExec or create), hooking only at higher layers leaves room for bypasses (e.g., via direct reflection on internal classes).

In theory, the deeper the hook point, the more robust the protection—provided performance overhead is acceptable. Unfortunately, the third layer (C/C++ native implementations) cannot be hooked from a Java Agent.

Thus, the optimal compromise is to hook at the second layer: the native methods themselves, using JVM instrumentation features that support native method prefixing.

• Unix-like systems:

JDK ≤ 8: java.lang.UNIXProcess.forkAndExec

JDK ≥ 9: java.lang.ProcessImpl.forkAndExec

• Windows systems (no version difference)

java.lang.ProcessImpl.create