In this series of articles I describe how I built a DWARF parser without JNI, without Unsafe, and without leaving Java, using the new FFM APIs…and it’s fast.
What is DWARF?
Debuggers don’t magically know where your variables live or which source line corresponds to the current instruction. That information comes from DWARF — a rich binary format embedded inside native executables.
DWARF is a debug information format that is built into binaries by compilers to support source-level debugging. It is language-agnostic and can be used by many language compilers. Documentation of the DWARF standard can be found at https://dwarfstd.org/
DWARF is the information that allows debuggers and tools to map lines of code to execution addresses and back again. If you want to set a breakpoint on a line of code, DWARF lets you map that line of code to an address in memory on which to set the breakpoint. Conversely, when you are debugging and hit a breakpoint, you want your debugger to show you the line of code you hit. DWARF lets you map the breakpoint address back to the line of code.
It also provides information about variables, functions, their arguments, their types and their locations for when you want to inspect variables, or stack traces.
While these posts don’t go into great detail about DWARF, some knowledge of it is required, and will be explained along the way.
Parsing it in Java
Typical approaches to writing libraries in Java that work with binary content often take the more object oriented approach that leads to performance issues in terms of memory and speed as the binary format is read and converted into classes. This parser will be using a no-copy approach where we read directly from the source binary data. This means we don’t materialise the source data into classes, but provide lightweight views on the underlying data directly so developers can still work with an object interface without having to provide a complete object graph.
Using the Foreign Function & Memory API
We’ll do it using the FFM APIs finalized in Java 22, which provide a modern alternative to many traditional JNI use cases to interoperate with native code and memory outside of the Java heap. One of its key features is the ability to call native libraries directly from Java without JNI or third-party bindings. However, for this project, the focus will be on its ability to interop with native memory, specifically file mapped memory.
FFM introduces a few new concepts that we will touch on over the course of these articles:
- Arena - Used to manage the lifecycle of off-heap native memory
- MemorySegment - A contiguous block of native memory that we can read/write to.
- MemoryLayout - A template or schema used to define the structure of data in memory. They can be as simple as numerical values, arrays, structures or unions.
- VarHandle - a strongly typed accessor that lets us read/write data from the MemorySegment according to the defined MemoryLayout instance.
The ELF File
DWARF exists in binary format in files, often in the same executable, if it was compiled with debug information enabled. DWARF data can be used in many file formats on different platforms but as I’m on Linux and used to working with ELF, we’ll start there.
The ELF file has a header starting at position 0, that has some useful details that can be seen by running readelf -h <executable>. Of particular interest is the section header information, which contains the number of entries, the size and the starting offset of the section header data in the file. These sections are not all DWARF related and many are present when debug information is not included in the compilation.
There is plenty of documentation about the ELF format and ELF Files.
Here’s the ELF header for a simple Hello World C program.
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: EXEC (Executable file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x400380
Start of program headers: 64 (bytes into file)
Start of section headers: 38896 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 13
Size of section headers: 64 (bytes)
Number of section headers: 39
Section header string table index: 38
This header tells us about the structure of the ELF file and where to find the different components and what they are like. The program headers are used by the OS to load the executable into memory, and the section headers describe the different sections in the file which contain code, data, debug information and so on.
In the ELF header you can see the section header string table index value which indicates which section contains all the string names of the section headers.
Our section headers are at location 38896 which means we want to jump to that location and read in the section headers which are 64 bytes in size. Again, using readelf we can get a good look at the sections in the executable:
$ readelf -SW ~/hello
There are 39 section headers, starting at offset 0x97f0:
Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al
[ 0] NULL 0000000000000000 000000 000000 00 0 0 0
[ 1] .note.gnu.build-id NOTE 0000000000400318 000318 000024 00 A 0 0 4
[ 2] .init PROGBITS 000000000040033c 00033c 00001b 00 AX 0 0 4
[ 3] .plt PROGBITS 0000000000400360 000360 000020 10 AX 0 0 16
[ 4] .text PROGBITS 0000000000400380 000380 0000fb 00 AX 0 0 16
[ 5] .fini PROGBITS 000000000040047c 00047c 00000d 00 AX 0 0 4
...
...
...
[24] .data PROGBITS 0000000000403008 002008 000004 00 WA 0 0 1
[25] .bss NOBITS 000000000040300c 00200c 000004 00 WA 0 0 1
[26] .comment PROGBITS 0000000000000000 00200c 00002e 01 MS 0 0 1
[27] .annobin.notes PROGBITS 0000000000000000 00203a 00014f 01 MS 0 0 1
[28] .gnu.build.attributes NOTE 0000000000405010 00218c 000144 00 0 0 4
[29] .debug_aranges PROGBITS 0000000000000000 0022d0 000030 00 0 0 1
[30] .debug_info PROGBITS 0000000000000000 002300 000095 00 0 0 1
[31] .debug_abbrev PROGBITS 0000000000000000 002395 00004d 00 0 0 1
[32] .debug_line PROGBITS 0000000000000000 0023e2 0000ec 00 0 0 1
[33] .debug_str PROGBITS 0000000000000000 0024ce 005718 01 MS 0 0 1
[34] .debug_line_str PROGBITS 0000000000000000 007be6 0001be 01 MS 0 0 1
[35] .debug_macro PROGBITS 0000000000000000 007da4 0013d1 00 0 0 1
[36] .symtab SYMTAB 0000000000000000 009178 000330 18 37 18 8
[37] .strtab STRTAB 0000000000000000 0094a8 0001aa 00 0 0 1
[38] .shstrtab STRTAB 0000000000000000 009652 000198 00 0 0 1
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
L (link order), O (extra OS processing required), G (group), T (TLS),
C (compressed), x (unknown), o (OS specific), E (exclude),
D (mbind), l (large), p (processor specific)
Note:
- From the ELF header, the section header string table is in section 38, and there it is at the end.
.shstrtab - The
.debug_infosection is the one we are most interested in at the moment. It contains all the debug information. - You can see the offset column for where in the file the section lives. If we hex dump the file from 0x9652 (
.shstrtab) we can see the null terminated names of the section headers above:
$ hexdump ~/hello -s 0x009652 -n 0x000198 -c
0009652 \0.symtab \0.strtab
0009662 \0.shstrtab \0.note
0009672 .gnu.build-id \0.i
0009682 nit \0.text \0.fini \0
0009692 .interp \0.gnu.has
00096a2 h\0.dynsym \0.dynst
00096b2 r\0.gnu.version \0.
...
...
Note that the first entry is effectively an empty string starting with \0. This pattern also occurs a lot in DWARF structures where the first item is a null or empty version of whatever structure the table contains. It’s so the DWARF content can have an effectively null item at index 0 and the parser doesn’t need to detect it, it will just process an empty structure.
Now we know where the DWARF data lives inside an ELF executable, before we can parse DWARF sections, we need to read and interpret the ELF file. In the next article we’ll build an ELF reader using Java’s Foreign Function & Memory API, starting with the ELF header.