Skip to content

Getting Started with Forensics

Tracee has a unique feature that lets you capture interesting artifacts from running applications, using the --capture flag.

sudo ./dist/tracee --help capture
sudo ./dist/tracee --capture xxx

Tip

All captured artifacts are saved in Tracee's "output directory", which can be configured using --capture dir:/path/to/dir. You may also use --capture clear-dir if you want contents of the destination directory to be cleared every time you execute tracee.

Artifacts Types

Tracee can capture the following types of artifacts:

  1. I/O Files

    Anytime a file is being written to and/or read from, the contents of the file will be captured. I/O files can be filtered using 3 optional filters: 1. path - prefix of the file written/read. Up to 3 path filters can be provided per capture type. 2. type - file's type can be pipe, socket or regular. 3. fd - standard FD, one of the following: stdin, stdout and stderr.

    write example

    sudo ./dist/tracee \
       --output json \
       --filter comm=bash \
       --filter follow \
       --output option:parse-arguments \
       --capture dir:/tmp/tracee/ \
       --capture write='/tmp/*'
    

    Note

    Using file capture without filter name will be path by default. Hence, --capture write='/tmp/* is the same as --capture write:path='/tmp/*.

    echo write testing 123 > /tmp/testing.txt
    
    {"timestamp":1657321167356748797,"threadStartTime":620311624458929,"processorId":7,"processId":2578238,"cgroupId":1,"threadId":2578238,"parentProcessId":2578237,"hostProcessId":2578238,"hostThreadId":2578238,"hostParentProcessId":2578237,"userId":1000,"mountNamespace":4026531840,"pidNamespace":4026531836,"processName":"bash","hostName":"fujitsu","containerId":"","containerImage":"","containerName":"","podName":"","podNamespace":"","podUID":"","eventId":"722","eventName":"security_file_open","argsNum":6,"returnValue":0,"stackAddresses":null,"syscall":"openat","contextFlags":{"containerStarted":false,"isCompat":false},"args":[{"name":"pathname","type":"const char*","value":"/tmp/testing.txt"},{"name":"flags","type":"string","value":"O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE"},{"name":"dev","type":"dev_t","value":271581185},{"name":"inode","type":"unsigned long","value":1966101},{"name":"ctime","type":"unsigned long","value":1657321027326584850},{"name":"syscall_pathname","type":"const char*","value":"/tmp/testing.txt"}]}
    {"timestamp":1657321167356729582,"threadStartTime":620311624458929,"processorId":7,"processId":2578238,"cgroupId":1,"threadId":2578238,"parentProcessId":2578237,"hostProcessId":2578238,"hostThreadId":2578238,"hostParentProcessId":2578237,"userId":1000,"mountNamespace":4026531840,"pidNamespace":4026531836,"processName":"bash","hostName":"fujitsu","containerId":"","containerImage":"","containerName":"","podName":"","podNamespace":"","podUID":"","eventId":"257","eventName":"openat","argsNum":4,"returnValue":3,"stackAddresses":null,"syscall":"openat","contextFlags":{"containerStarted":false,"isCompat":false},"args":[{"name":"dirfd","type":"int","value":-100},{"name":"pathname","type":"const char*","value":"/tmp/testing.txt"},{"name":"flags","type":"string","value":"O_WRONLY|O_CREAT|O_TRUNC"},{"name":"mode","type":"mode_t","value":438}]}
    

    Note

    You can read captured files written at /tmp/tracee/out:

    sudo cat /tmp/tracee/out/host/write.dev-271581185.inode-1966101
    

    write testing 123
    

    read example

    sudo ./dist/tracee \
       --output json \
       --filter comm=bash \
       --filter follow \
       --output option:parse-arguments \
       --capture dir:/tmp/tracee/ \
       --capture read:type=pipe \
       --capture read:fd=stdin'
    
    echo read testing 123 | cat
    
    {"timestamp":1685285181028166900,"threadStartTime":1685285181026565700,"processorId":1,"processId":182934,"cgroupId":1,"threadId":182934,"parentProcessId":147428,"hostProcessId":184128,"hostThreadId":184128,"hostParentProcessId":148293,"userId":0,"mountNamespace":4026532277,"pidNamespace":4026532279,"processName":"cat","hostName":"Alon-Zivony","container":{},"kubernetes":{},"eventId":"720","eventName":"vfs_read","matchedPolicies":[""],"argsNum":5,"returnValue":17,"syscall":"read","stackAddresses":null,"contextFlags":{"containerStarted":false,"isCompat":false},"args":[{"name":"pathname","type":"const char*","value":""},{"name":"dev","type":"dev_t","value":12},{"name":"inode","type":"unsigned long","value":174033},{"name":"count","type":"size_t","value":131072},{"name":"pos","type":"off_t","value":0}]}
    {"timestamp":1685285181028267200,"threadStartTime":1685285181026565700,"processorId":1,"processId":182934,"cgroupId":1,"threadId":182934,"parentProcessId":147428,"hostProcessId":184128,"hostThreadId":184128,"hostParentProcessId":148293,"userId":0,"mountNamespace":4026532277,"pidNamespace":4026532279,"processName":"cat","hostName":"Alon-Zivony","container":{},"kubernetes":{},"eventId":"720","eventName":"vfs_read","matchedPolicies":[""],"argsNum":5,"returnValue":0,"syscall":"read","stackAddresses":null,"contextFlags":{"containerStarted":false,"isCompat":false},"args":[{"name":"pathname","type":"const char*","value":""},{"name":"dev","type":"dev_t","value":12},{"name":"inode","type":"unsigned long","value":174033},{"name":"count","type":"size_t","value":131072},{"name":"pos","type":"off_t","value":0}]}
    

!!! Note You can read captured files read at /tmp/tracee/out:

sudo cat /tmp/tracee/out/host/read.dev-12.inode-176203

    ```text
    read testing 123
    ```
  1. Executed Files

    Anytime a binary is executed, the binary file will be captured. If the same binary is executed multiple times, it will be captured just once.

    sudo ./dist/tracee \
       --output json \
       --filter comm=bash \
       --filter follow \
       --output option:parse-arguments \
       --capture dir:/tmp/tracee/ \
       --capture exec
    
    /bin/ls
    
    {"timestamp":1657322300531713371,"threadStartTime":620311624458929,"processorId":21,"processId":2578238,"cgroupId":1,"threadId":2578238,"parentProcessId":2578237,"hostProcessId":2578238,"hostThreadId":2578238,"hostParentProcessId":2578237,"userId":1000,"mountNamespace":4026531840,"pidNamespace":4026531836,"processName":"bash","hostName":"fujitsu","containerId":"","containerImage":"","containerName":"","podName":"","podNamespace":"","podUID":"","eventId":"56","eventName":"clone","argsNum":5,"returnValue":3331757,"stackAddresses":null,"syscall":"clone","contextFlags":{"containerStarted":false,"isCompat":false},"args":[{"name":"flags","type":"string","value":"CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID"},{"name":"stack","type":"void*","value":"0x0"},{"name":"parent_tid","type":"int*","value":"0x0"},{"name":"child_tid","type":"int*","value":"0x7fd7ce0d3a10"},{"name":"tls","type":"unsigned long","value":0}]}
    {"timestamp":1657322300534562489,"threadStartTime":620311624458929,"processorId":21,"processId":2578238,"cgroupId":1,"threadId":2578238,"parentProcessId":2578237,"hostProcessId":2578238,"hostThreadId":2578238,"hostParentProcessId":2578237,"userId":1000,"mountNamespace":4026531840,"pidNamespace":4026531836,"processName":"bash","hostName":"fujitsu","containerId":"","containerImage":"","containerName":"","podName":"","podNamespace":"","podUID":"","eventId":"3","eventName":"close","argsNum":1,"returnValue":0,"stackAddresses":null,"syscall":"close","contextFlags":{"containerStarted":false,"isCompat":false},"args":[{"name":"fd","type":"int","value":3}]}
    

    Note

    You will have a copy of each executed file written at /tmp/tracee/out:

    ldd /bin/ls
    

    linux-vdso.so.1 (0x00007ffca632c000)
    libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f9a930d5000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9a92ead000)
    libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f9a92e16000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9a93136000)
    
    ldd /tmp/tracee/out/host/exec.1657322052835478987.ls
    
    linux-vdso.so.1 (0x00007ffe337fb000)
    libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007feeb1fa5000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007feeb1d7d000)
    libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007feeb1ce6000)
    /lib64/ld-linux-x86-64.so.2 (0x00007feeb2006000)
    
    ```console
    sudo chmod +x /tmp/tracee/out/host/exec.1657322052835478987.ls
    /tmp/tracee/out/host/exec.1657322052835478987.ls
    
  2. Memory Files

    Anytime a memory unpacker is detected, the suspicious memory region will be captured. This is triggered when memory protection changes from Write+Execute to Write.

    sudo ./dist/tracee \
       --output none \
       --filter comm=bash \
       --filter follow \
       --capture dir:/tmp/tracee/ \
       --capture mem
    

    Note

    You may opt not to have any output from tracee with --output none command flag is given. This makes tracee to work in capture mode only.

  3. Network PCAP Files

    Anytime a network packet is delivered to a process, traced by tracee, this packet might be captured into one or multiple pcap files.

    Attention

    The default behavior when capturing network traffic is to capture ALL traffic, despite given event filters. If you want to make capture feature to follow the given event filters, like for example capturing DNS events only, then you have to provide --capture pcap-options:filtered argument in the command line. Then only net_packet_XXX events will be captured (IPv4, IPv6, TCP, UDP, ICMP, ICMPv6, DNS, HTTP, etc).

    A good way to test this behavior is to execute:

    sudo ./dist/tracee \
        --filter event=net_packet_ipv4 \
        --capture network \
        --capture pcap-options:filtered
    

    and observe a single pcap file for all ipv4 packets created:

    find /tmp/tracee/out/pcap/
    
    /tmp/tracee/out/pcap/
    /tmp/tracee/out/pcap/single.pcap
    

    You can select only dns packets, for example:

    sudo ./dist/tracee \
        --filter event=net_packet_dns \
        --capture network \
        --capture pcap-options:filtered
    

    and the file /tmp/tracee/out/pcap/single.pcap would only contain DNS related packets:

    find /tmp/tracee/out/pcap/
    
    /tmp/tracee/out/pcap/
    /tmp/tracee/out/pcap/single.pcap
    
    sudo tcpdump -n -r /tmp/tracee/out/pcap/single.pcap | head -2
    
    reading from file /tmp/tracee/out/pcap/single.pcap, link-type NULL (BSD loopback), snapshot length 262144
    16:53:48.870629 IP 127.0.0.1.55569 > 127.0.0.53.53: 33361+ [1au] A? www.uol.com.br. (43)
    16:53:48.870690 IP 127.0.0.1.55569 > 127.0.0.53.53: 25943+ [1au] AAAA? www.uol.com.br. (43)
    

    A great thing is that you may have multiple pcap files, divided by:

    1. single: a single pcap file containing all packets (the default)
    2. process: one file per process executed, ordered by host and container
    3. container: one file for the host and one pcap file per container
    4. per-command: one file per command executed (even if multiple times)

    and you can even have multiple ways at the same time. Example: a ping command is executed inside a container. You want to summarize captured traffic per container and per command. You will find the same captured data for that ping command inside commands/container_id/ping.pcap and inside containers/container_id.pcap.

    sudo ./dist/tracee \
        --filter event=net_packet_icmp \
        --capture network \
        --capture pcap-options:filtered \
        --capture pcap:process,container,command
    
    cd /tmp/tracee/out
    find pcap
    
    pcap
    pcap/commands
    pcap/commands/b86533d11f3
    pcap/commands/b86533d11f3/ping.pcap
    pcap/commands/host
    pcap/commands/host/sshd.pcap
    pcap/commands/host/zerotier-one.pcap
    pcap/commands/host/node.pcap
    pcap/commands/fd95a035ce5
    pcap/commands/fd95a035ce5/ping.pcap
    pcap/processes
    pcap/processes/b86533d11f3
    pcap/processes/b86533d11f3/ping_1261180_1663772450241192.pcap
    pcap/processes/host
    pcap/processes/host/node_186708_1573567360495399.pcap
    pcap/processes/host/node_1196826_1662656211119567.pcap
    pcap/processes/host/zerotier-one_7882_137007714376.pcap
    pcap/processes/host/sshd_1196773_1662654999660718.pcap
    pcap/processes/fd95a035ce5
    pcap/processes/fd95a035ce5/ping_1261163_1663769383806467.pcap
    pcap/containers
    pcap/containers/host.pcap
    pcap/containers/b86533d11f3.pcap
    pcap/containers/fd95a035ce5.pcap
    

    you can see the packets by executing tcpdump on any pcap file:

    tcpdump -r pcap/containers/b86533d11f3.pcap
    
    reading from file pcap/containers/b86533d11f3.pcap, link-type NULL (BSD loopback), snapshot length 65535
    02:52:00.524035 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 476, length 64
    02:52:00.533145 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 476, length 64
    02:52:01.525455 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 477, length 64
    02:52:01.535414 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 477, length 64
    02:52:02.526715 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 478, length 64
    02:52:02.536444 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 478, length 64
    02:52:03.528739 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 479, length 64
    02:52:03.538622 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 479, length 64
    
    tcpdump -r pcap/commands/b86533d11f3/ping.pcap
    
    reading from file pcap/commands/b86533d11f3/ping.pcap, link-type NULL (BSD loopback), snapshot length 65535
    02:52:00.524035 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 476, length 64
    02:52:00.533145 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 476, length 64
    02:52:01.525455 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 477, length 64
    02:52:01.535414 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 477, length 64
    02:52:02.526715 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 478, length 64
    02:52:02.536444 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 478, length 64
    02:52:03.528739 IP 172.17.0.3 > dns.google: ICMP echo request, id 5, seq 479, length 64
    02:52:03.538622 IP dns.google > 172.17.0.3: ICMP echo reply, id 5, seq 479, length 64
    

    Note

    Note that the same packets were written to 2 different pcap files: the pcap file describing the container b86533d11f3 (because it was executing a single process: ping) and the pcap file describing ANY ping command executed in that container (commands/b86533d11f3/ping.pcap).

    The format for the pcap filenames inside output_dir is the following:

    1. single:
      ./pcap/single.pcap
    2. processes:
      ./pcap/processes/container_id/process_comm/host_tid/task_starttime.pcap
    3. containers:
      ./pcap/containers/container_id.pcap
    4. commands:
      ./pcap/commands/container_id/process_comm.pcap

    Attention

    By default, all pcap files will contain packets with headers only. That might too little for introspection, since sometimes one might be interested in a few bytes of the captured packet (or event it all). Next item shows how to capture a specific packet payload size.

    In order to capture a specific payload size you may specify:

    sudo ./dist/tracee \
        --filter event=net_packet_tcp \
        --capture network \
        --capture pcap-options:filtered \
        --capture pcap:single,command \
        --capture pcap-snaplen:default
    

    To capture packet headers + 96 bytes of payload. Or replace default by:

    1. headers: capture up to L4 headers only
    2. max: full sized packets into pcap. WARNING: big pcap files.
    3. 256b, 512b, 1024b, ... (any number plus "b")
    4. 16kb, 32kb, 64kb, ... (any number plus "kb")

    when specifying a payload size, it refers to the payload AFTER the layer4 headers (and not the entire packet length).

  4. Loaded Kernel Modules

    Anytime a kernel module is loaded, the binary file will be captured. If the same binary is loaded multiple times, it will be captured just once.

    sudo ./dist/tracee \
        --output none \
        --filter comm=bash \
        --filter follow \
        --capture clear-dir \
        --capture module
    

    Captured module will be found in tracee destination directory, just like any other captured file would be:

    sudo ls /tmp/tracee/out/host
    
    module.dev-271581185.inode-4071826.pid-3668786.c8b62228208f4bdbf21df09c01046b73dd44733841675bf3c0ff969fbedab616
    

    AND, the captured module is an exact copy of the loaded module:

    sudo rmmod lkm_example
    sudo insmod /tmp/tracee/out/host/module.dev-271581185.inode-4071826.pid-3668786.c8b62228208f4bdbf21df09c01046b73dd44733841675bf3c0ff969fbedab616
    
    lsmod | grep example
    
    ```text
    lkm_example            16384  0
    
    sudo rmmod lkm_example
    

    you can even load/unload it.

    Note

    Example kernel module taken from this blog

  5. BPF programs

    Wheneber a BPF program is loaded, the BPF bytecode will be captured. This captured bytecode represents the BPF program as it was loaded by the loading process. It is basically the BPF section of the compiled ELF that loads the BPF program, which contains the program instructions themselves. You can disassemble the bytecode with the help of binutils-bpf package and the following command line: objdump -D -b binary -m bpf <path>

    $ sudo ./dist/tracee-ebpf \
       --output none \
       --filter comm=bash \
       --filter follow \
       --capture clear-dir \
       --capture bpf
    

    Captured bpf bytecode will be found in tracee destination directory, just like any other captured file would be:

    $ sudo ls /tmp/tracee/out/host
      bpf.name-test_prog.pid-3668786.c8b62228208f4bdbf21df09c01046b73dd44733841675bf3c0ff969fbedab616
    
    The hex value after the last "." is the hash of the bpf bytecode.