Broadcom eCos | Writing a device profile for bcm2-utils
In this blog post we’ll dive into jclehner’s bcm2-utils tools and perform the following steps:
dump an unknown bootloader with bcm2dump
reverse engineer specific sections of the booloader
write a device profile for bcm2dump
dump the NAND flash and extract the eCos firmware
dump the SPI flash and analyze non-vol settings
From there, we will patch non-vol settings to enable console access, flash it and then adapt the console section of our initial bcm2dump profile.
The device under test is the Siligence TCG300 from Orange, a white-branded ASKey device. The device sports three 4 pins pinouts labelled UART0, UART1, UART2.
UART0 is live while the others are not.
The pins setup for reference:
From early boot information, we see that the device bootloader is unlocked. You can see that from the Enter ‘1’, ‘2’, or ‘p’ prompt, allowing to enter the bootloader menu by pressing ‘p’.
But even though the bootloader is unlocked, we cannot access the cable modem console given that console input/output has been explicitly disabled in non-volatile storage:
To read the flash content, we have to instrument the bootloader code by writing a custom profile for bcm2-utils.
Dumping the bootloader
Quoting bcm2-utils documentation:
An easy way to locate the bootloader is to jump to an arbitrary location in RAM, and then study the exception handler’s output. Jumping to a random address is one way to crash your device, but to be safe, you could write an opcode to RAM that will cause a crash, and then jump to that location. Something like sw $zero, 0($zero) (0xac000000) is always a safe bet.
Let’s do exactly that !
The most important info here is the value from register $ra, but we can also see many other references to 0x83f8XXXX, so it’s safe to assume that the bootloader is loaded somewhere around this address.
Restart the device, go into the main menu again, and we can fire up bcm2dump to dump the bootloader code from ram. The bootloader is usually very small, around 64k. To be safe, we’ll dump 128k before and after 0x83f80000:
I consider that you can connect a USB to UART adapter on your Linux box to run this bcm2dump command:
Cleaning Things Up
Looking at dumped content, the beginning is mostly 0xff up to offset 0x00020000:
The data section starts at offset 0000d690 (0x83f8d690):
Then at offset 0x00016000 we clearly have nothing more:
Let’s remove the 0x00020000 (131072 in decimal) first bytes with dd, while stopping at offset 0x00016000 (90112 in decimal). This way we get a clean bootloader image without any garbage data before or after.
Now that we have a clean bootloader image, it’s time to write our initial profile.
Writing the profile
We can start with what we know from the boot logs and initial information gathering:
the device name
the firmware image ‘signature’ (psig)
the baud rate
the broadcom chipset model
In profiledef.c, you can append the following value:
Flash Partitions Layout
Now we need to define each memory space layouts (ram, nvram, flash) in the .spaces section. To do so, we print the flash partition by typing ‘p’ in the bootloader menu:
We have 9 partitions but they’re not located in the same chip. bootloader, permnv, and dynnv are located onto the SPI flash (nvram in bcm2utils lingo), while all the others are located onto the NAND flash (flash in bcm2utils lingo). Let’s convert that information into our bcm2utils profile:
Profile Auto Detection
bcm2dump supports profile auto-detection. For bcm2dump to be able to auto-identify your device, you need to tell it whether you expect it to launch the detection when in bootloader prompt (BCM2_INTF_BLDR) or console prompt (BCM2_INTF_BFC). You then provide a .magic value. This value is a tuple holding a memory address and expected value. bcm2dump will read the value at the given memory location and check it against the expected value. If they match, the profile is detected.
The .intf definition tells bcm2dump what to look for when connecting to the console.
You have to go through strings in the firmware to find good candidates. Here’s the one I used, displayed in Ghidra:
The rwcode and buffer values are copy/pasted from Compal and NetMASTER profiles. They worked perfectly.
Flash Read Functions Definition
bcm2dump can instrument existing bootloader code for faster dumping of both SPI and NAND flash chips. To get that feature working for our profile, we need to identify the functions in charge of reading from these chips, along with their function signatures.
Let’s see how we can identify these functions.
Loading A Bootloader Image in Ghidra
Remember the bootloader we extracted at the beginning ? Now it’s time to load it in Ghidra as a MIPS big endian 32 bits raw binary. The load address is precisely 0x83f80000 (remember that we dumped memory starting from address 0x83f60000 but that we had 0x20000 bytes of garbage).
Note: The architecture and endianness were derived from an open source bootloader for BCM3384 chips that mentions 32-bit MIPS BE in its documentation.
We can identify numerous functions from calls to verbose logging functions.
The one we’re interested in is NandFlashRead at offset 0x83f83e9c.
Automating Function Identification
Most bootloaders I analyzed still have verbose logging and we can use that to our advantage. The process is dead simple:
identify log call
extract function name from the log call
rename the function where log function is called with the extracted name
I wrote the script below using radare2/r2pipe but feel free to re-implement it in your language of choice with your favorite SRE tool.
Running the script on the Siligence bootloader will give you something along these lines:
Thanks to the information gained with the script, I was able to create the profile section below. I specified a read function for the NAND flash at address 0x83f83e9c and that this function signature is buffer, offset, length (BCM2_READ_FUNC_BOL, BOL means Buffer Offset Length). I also specified a read function for the SPI flash located at address 0x83f81324, with an offset, buffer, length signature (BCM2_READ_FUNC_OBL).
Once that’s done, compile the new profile and run the tool to see if the device is detected:
If that works, we’re ready to dump the NAND content. First bcm2dump will patch the code in memory and then trigger calls to dump the flash over serial:
Dumping SPI Flash
Dumping dynamic settings is also super easy:
We can read it using bcm2cfg:
Bypassing a Disabled Console Prompt
If you remember the boot logs, we cannot access the device console because it’s been explicitly disabled in the non-vol settings:
I explored three avenues when trying to bypass this protection:
Patching the firmware code
Patching the permnv settings
Patching the dynnv settings
Patching Firmware Code
The initial idea was to patch the firmware image to bypass the console enable checks. The check is performed at 0x80166508 and we should replace it with an unconditional jump.
We can see the beq v0,zero instruction below (10 40 00 23):
We patch it with an unconditional jump:
Once it is patched, we repack it by reproducing the image format:
We can serve the firmware over TFTP:
Press ‘g’ menu in bootloader to load the image and execute from RAM. The image loads and get decompressed but then we have a crash for some unknown reason.
If we look at raw data from the permnv, we see that the data is all wrong. The length of the bfc section is supposed to be 0x09 bytes, but if it is, it is overlapping with the length (0x3d) of the next section (MLog).
Compared to a valid format taken from dynnv:
Given the weird format, it is highly unlikely the firmware reads data off permanent non-vol storage to check if console access is enabled or not. Most likely this data is used as a skeleton to rebuild the dynamic storage section during factory reset. Moving on.
First, let’s dump dynnv from the SPI flash using bcm2-utils:
We can see that serial_console_mode is set to disabled:
Let’s rewrite it:
Now that we have a modified dynnv partition, it’s time to write it back to the device. The problem here is that bcm2dump does not support (yet) writing back to nvram or flash from the bootloader menu. I filed an issue and will most likely work on it in the near future.
Edit: It turns out I was wrong and bcm2dump does support writing back to nvram. I was just missing the right flags. I’ll edit this at some point in the future, writing back using SPI protocol worked well anyway :)
In the meantime, I simply plugged into the SPI flash with an 8-pin SOIC clip. The chip is a Macronix MX25L8006E, with the following pinout:
We read the content from the SPI flash with flashrom:
The format is exactly the one from the map, so we can rebuild a modified version
Closer examination of the flash image shows that there are a number of repeated copies of the configuration following the initial one.
Let’s take a look by grepping for the bfc configuration section’s magic (“CMAp”):
Right, so we have multiple copies and only the first one got modified by bcm2cfg.
I - like a dumbass - initially patched the values manually without taking care of the CRC of each dynnv copy. Which ended up in multiple reflashing, reboots, and overall confusion.
This is the kind of greeting you get when CRC is wrong:
A smarter approach is to search for dynnv magic header (‘\xff’ repeated 202 times):
The script identifed three different copies of dynnv within the dynnv partition:
From that, we will cut the dynnv in separate sections and edit each of them with bcm2cfg:
We re-flash the SPI with /tmp/tcg3000.modified.spi, reboot the device. And now we have a working shell !
Edit: the matcher script has to be edited for your own needs/device. After a factory reset of the TCG300, the structure was different and held 10 copies of dynamic settings, each exactly 23830 bytes long.
Expanding the Profile (CM console)
As a final touch, let’s expand our profile to add auto-detection for our device when it’s at the console prompt.
First, we need to pull the firmware image.
Then we can extract it using ProgramStore. To do so, you’ll need to compile ProgramStore from Broacom aeolus repository. Then simply run it in extraction mode:
Then you can load it in Ghidra as a raw MIPSBE 32bits binary with load address 0x80004000.
Ideally, you should look for a string that’s printed when running the ‘version’ command. In this case, we’ll look for the revision value (‘5.7.1mp2’).
Looks like it’s stored at offset 0x80f62b18:
The version definition is similar to the one I wrote for the bootloader interface:
The device is now auto-detected in console mode and you can also dump memory from console mode (although way slower than in bootloader mode):
At this point I started fuzzing the device’s web interface and noticed the following output:
After that, the device fully rebooted. This is strange behavior because eCos devices tends to print out a stack trace with a list of running threads, register values, and memory dump when a crash happen.
We are logged into the ‘Cable Modem’ shell (CM) but could it be that the device also expose a ‘Router Gateway’ (RG) shell ? I switched my connections from UART0 to UART2 and there it was !
Different, but similar. The fact that a single eCos kernel run specific code within a specific CPU core is super interesting, but I won’t cover this right now.
Over the course of this article, I explained how to approach an unknown device running Broadcom eCos, dump its bootloader, instrument it to dump the full NAND and SPI flash, patch non-volatile settings to obtain console access, and discovered a second console access.
If you have any questions, feel free to get in touch via email or Twitter.