Multi-FPGA Embedded System Debug

The Customer was struggling with a mission critical, embedded design, hosted on a main FPGA board Microblaze soft processor design interfaced with eight daughter cards. A TCP/IP stack provided connectivity to the PC with data collected from eight infrared sensors on each of the daughter cards. The traditional SDK based debugging had run out of steam as months had passed with no viable leads on the cause of a random operational bug. Utilizing Xilinx® Chipscope™, I was able to locate a randomly pulsing signal used in an asyncronous FSM reset. Driving to root cause, I examined the board schematic and discovered a level translator employed to supply a 'board present signal' driving an FPGA pullup input. Forbidden by the translator's device datasheet, the output driver could not overcome the weak pullup in the FPGA. The original (two) FPGA HDL authors, the software engineer and the board design engineer had completely missed the subtle bug caused by this connection. Also, upgraded the socket connection from LwIP to the commercial Treck stack. Implemented FPGA source code control using Subversion and took the design through eight major releases.