A Lander on Mars Kept Rebooting Itself
July 1997. Mars Pathfinder had stuck the landing, Sojourner was rolling, and the mission was a front-page triumph. Then the lander started resetting itself. Not once: repeatedly, seemingly at random, each reset killing a day's science while the spacecraft climbed back to a safe state. The press called it a software glitch. What it actually was has become the most famous concurrency bug in engineering history, and every RTOS engineer should be able to explain it on a whiteboard.
The canonical account comes from Mike Jones' widely circulated "What really happened on Mars" write-up and the detailed response from Glenn Reeves, who led the Pathfinder flight software team at JPL. The primary sources are worth reading in full (Reeves' account, archived at Microsoft Research), because the story is better than the legend.
The Setup: Three Tasks and One Mutex
Pathfinder's flight software ran on VxWorks with preemptive priority scheduling. Three tasks matter here.
A high-priority bus management task moved data through the spacecraft's "information bus," a shared area guarded by a mutex. A low-priority meteorological task published weather data to that same bus, briefly taking the same mutex. And a medium-priority communications task ran long, doing radio work, with no interest in the mutex at all.
The fatal interleaving: the low-priority weather task grabs the mutex. The high-priority bus task wakes, needs the mutex, and blocks, politely waiting for the weather task to finish its brief critical section. Then the medium-priority communications task wakes and preempts the weather task, because medium outranks low, and runs for a long time.
Now walk the chain. The high-priority task is waiting on the low-priority task, which is waiting on the CPU, which is held by the medium-priority task. The system's most important work is effectively running at its least important priority. That's priority inversion: not a deadlock, everything would eventually finish, but "eventually" is not a word hard real-time systems get to use.
The Watchdog Tells the Truth
Pathfinder had a watchdog: if the bus management cycle didn't complete on schedule, the system concluded something was catastrophically wrong and reset the spacecraft. It was right to. From the watchdog's perspective, the highest-priority task in the system had silently stopped meeting its deadline.
So the resets weren't the bug. The resets were the safety system correctly reporting the bug. This distinction matters, and it's the part teams skip when they read the summary: Pathfinder was designed to fail loudly rather than compute quietly with a wedged bus. On Mars, loud failure saved the mission.
The Fix Was One Boolean, Uploaded From Earth
VxWorks mutexes had an option the team had left disabled: priority inheritance. With inheritance on, a task holding a mutex temporarily inherits the priority of the highest task blocked on it. The weather task, holding the mutex the bus task needed, would have been boosted to the bus task's priority, and the communications task could never have preempted it mid-critical-section.
JPL reproduced the failure on a ground replica, using VxWorks' tracing to catch the interleaving, then changed the mutex initialization flag and uploaded the patch to Mars. One boolean. The resets stopped.
Two engineering lessons hide in that patch. First: they could reproduce it, because they kept a faithful testbed and the RTOS had real tracing. The bug had actually appeared in pre-launch testing but was rare and unreproduced, filed under "we'll watch it." Second: the fix existed in the kernel all along. Knowing your RTOS's mutex semantics isn't trivia. It's flight software.
What To Do About It in Your Own Firmware
Turn on priority inheritance for any mutex shared across priority levels. In FreeRTOS, mutexes (as opposed to binary semaphores) implement priority inheritance by default, which is exactly why the FreeRTOS documentation tells you to use a mutex, never a semaphore, for mutual exclusion. The alternative discipline, priority ceiling, raises the holder to a pre-computed ceiling the moment it takes the lock, and is favored in some certified systems because its worst case is easier to analyze.
Keep critical sections short and non-blocking. Inheritance bounds the inversion; it doesn't make a bloated critical section acceptable.
And respect your watchdog. If it's firing, the temptation is to lengthen the timeout. Pathfinder's team treated the reset as data, traced the scheduler, and found a three-task interleaving forty million miles away. The watchdog wasn't the problem then, and it probably isn't in your system either.
Frequently Asked Questions
Is priority inversion a VxWorks bug? No. It's inherent to priority scheduling plus shared locks, on any kernel. VxWorks provided the solution as a documented option; the configuration chose not to use it. The same trap exists today in FreeRTOS, Zephyr, ThreadX, and your homegrown scheduler, which almost certainly handles it worse.
Why not just never share a mutex across priorities? Sometimes you can architect it away with queues and a single owner per resource, and that's genuinely the cleaner design. But some resources are intrinsically shared, and pretending otherwise just moves the contention somewhere less visible.
Did the bug threaten the mission? It cost science time, not the spacecraft; the resets were safe by design. The reason it's famous isn't the damage. It's that the failure was subtle, the diagnosis was brilliant, the fix was one flag, and the whole story is documented by the people who lived it. Concurrency bugs rarely come with this good a teacher.
