Filed under: Embedded, Engineering

So I don't carry a smartphone with me for a mixture of reasons. Privacy obviously, I don't think at this point that needs spelling out. Environmental concerns are high up for me, though. Having to buy a new device every few years is not environmentally friendly. When I went looking for an alternative I looked at (and tried) a lot of things, but landed on a weird option.

The ClockworkPi uConsole looks cool. I get told it looks "cyberpunk" a lot. Thats... nice, but not the point. It is all metal, made of modular boards and has published schematics. It is powered by 2 18650 batteries and even has space in the case for an expansion board you can buy or build yourself. Durability, modularity, expandability. It is a really nice device.

This post isn't about the uConsole itself, but about an addon I built for and the challenges getting that addon working reliably. While ClockworkPi ships an internal 4G modem for the uConsole, and while I was able to get my hands on one, I wound up not using it. The T-Mobile 4G sunset is coming up next year. I don't like doing projects that only last a year or so, and getting the 4G modem working reliably would be a project. Also, I needed the 2nd USB port on the expansion connector exposed so I could connect my camera to it.

My solution was to get a 5G USB modem and use that velcro'd to the back of the case. To do this, though, I wanted a USB port that I could turn on and off in software. I don't need the modem anywhere I have wifi. The first version of this board was a bunch of parts hot glued to the blank 4G card that ships with the uConsole. It worked... not well, but it proved out that the idea was feasible if I could get past one error message.

[21351.442444] usb usb1-port1: disabled by hub (EMI?), re-enabling...

I would get runs of 10s of this message and afterword the USB hub on the uConsole motherboard would be unstable. Some connected devices like the keyboard would recover after a timeout, others (like the camera or my lapdock) would not work again until a power cycle. Clearly something had happened to make the hub unstable. Finding out what that was, and fixing it, took me down a months long rabbit hole that I will now describe.

The first and most obvious problem was construction. The prototype was a mess of wires, break out boards and hot glue. After the first tests I got to work designing a proper PCB. It needed to be thin to fit into the uConole's expansion connector, so I use OSH Park's 0.8mm 2oz process. That worked out great, the boards fit perfectly. I kept the design simple, 90 ohm diff paris for the external USB connector and camera and a TPS61023 breakout board to provide power to the external USB. I added this because it will fall back to pass through mode if the input is 5v, but boost if there is a dip. I had thought that a dip on the supply going to the modem might account for the errors, so I wanted to make sure that wasn't the case.

The first board was better, I got fewer errors, but they did not go away. When they happened I would still loose the camera, so the instability was still there. I tried different USB-C cables connected to the modem but that wasn't it. Then I made a fateful discovery, if I connected the modem to the uConsole's USB-A port I still got the error. This ruled out my board, but not the modem or the uConsole. If I connected the modem to another computer, though, I couldn't repoduce the error there. That pointed to the uConsole.

I spent a lot of time looking at the schematic. A lot of time. The problem wasn't obvious at all, but I did find it. The USB hub on the uConsle is a GL850G. This chip has it's own LDO for generating it 3.3v power rail from an existing 5v rail. This is useful if it is being used an external hub, but for some reason it is being powered that way on the uConsole even though the PMIC generates a 3.3v rail. Worse, the 5v rail doesn't have any dedicated capacitor feeding that LDO. Any small dip on the rail will push the LDO out of regulation. At first I experimented with added an 220uF cap to the 5v rail right at the hub. This improved the situation a lot, I got fewer errors and they didn't always result in the hub becoming unstable. I thought I solved it, until eventually the camera disappeared.

I knew I was on to something though, so I removed that mod and lifted pin 23 on the hub (the 5V input) and powered the hub from the 3.3v rail generated by the PMIC by connecting the supply side of C505 and C143 together. Thats actually a really small run of magnet wire on top of the board. Again the situation improved. I went all through SakruaCon with a working modem, but not a working camera. It would still drop out.

Solving layered problems starts with figuring out that the problem is in fact layered. Its easy to see the results and think the lack of progress is due to not going far enough when instead the real problem is somewhere else entirely. To be sure, the USB hub power was a problem and I encourage all uConsole owners facing USB stability issues to make that mod. It was not, however, the only problem.

I didn't include an ESD diode pack on the initial design of my board. This was an oversight due to rushing too much, but when I started on a redesign I added one. ESD could definatly cause the USB hub to go unstable, and since this thing lives it's life in my purse or pocket its going to encounter static. I got this new board in and to my surprise things got worse. It would work fine and then I would get a raft of that same error. I was watching dmesg the other evening and found the other culprit, if I flexed the connector on this new board just a little it would produce errors.

This may not have been a problem on the first PCB. These USB-C connectors are hard to solder down with just an iron and hot air. I got another USB-C cable that didn't stick out from the case and have been testing with that for the last 3 days without issue. No error messages, no drop outs, camera has worked the whole time.

This wasn't one problem, it was multiple problems that all generated the same error message. Ironically, EMI wasn't even one of the problems even though the kernel message hints at it. Working these kinds of problems its easy to be led astray or get tunnel vision on one aspect of the problem an miss other factors that might be leading to the fault. We have to step back and look at the whole system periodically to regain the context the fault happens in.