RaspberryPi Consistently Running out of Memory (LND/Raspibolt)

Short answer: Check if you have a monitor plugged into the pi. If so, unplug that sucker. If not, read on for a few troubleshooting techniques.

I have a LND node running on a Raspberry Pi and I love that little thing, I really do — but it can be quite temperamental. It all started when I tried messing with the fstab file. I wanted the pi to automatically remount the hard drive in the event of a disconnection. Somewhere along the way I messed up big time because I completely locked myself out of it. Since I couldn’t get the pi to fully boot up and get on the network I did my troubleshooting with a keyboard and monitor.

After I restored the pi I thought all was well, but it wasn’t. For the next few months my LND node was crashing a lot. I couldn’t unlock the wallet at all. lncli unlock kept producing one of two errors:

[lncli] rpc error: code = Unavailable desc = transport is closing

[lncli] rpc error: code = Unavailable desc = grpc: the connection is unavailable

Further examination into the LND logs ( .lnd/logs/bitcoin/mainnet/lnd.log) confirmed it was running out of memory every single time.

Try Increasing the Swap Space

Apparently LND executes some memory intensive scrypt code on startup so it wasn’t out of the realm of possibility that it needed more memory. I used free -h to see the amount of free and used memory in the system and decided to increase the swap space to 250m.

I followed these instructions and did the following:

dd if=/dev/zero of=/swap bs=1M count=250
mkswap /swap

Add this to /etc/fstab (nano /etc/fstab)
/swap swap swap defaults 0 0

swapon -a

It worked! Well, for a little while. It wasn’t long before the OOM errors began to return.

free -h
total used free shared buff/cache available
Mem: 927M 839M 30M 2.8M 57M 36M
Swap: 349M 282M 67M

Even with 250m swap space things still weren’t looking great.

Use ps aux to Find the Offending Process

After a lot of searching, I finally realized was not LND, it was me. I think this is important. Oftentimes, I already have all the tools I need to fix a problem, but the first step is admitting it was actually caused by something I did. Admittedly, I spend a lot of time denial-searching on the Internet, looking for solutions to issues I couldn’t possibly have created.

The ps aux command shows all the processes that are currently running and how much CPU and memory each uses. That’s when I saw this:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

pi 761 5.2 57.9 741972 549828 ? Sl Sep13 8102:03 lxpanel --profile LXDE-pi

Holy moly! What in the world is lxpanel and why is it using OVER 50% of my precious memory?!

Unplug the Damn Monitor

It turns out lxpanel is some GUI thing for the pi. Notice how the lxpanel process started September 13, the day I posted about fixing the messed up fstab file? Yeah, no good deed goes unpunished. The second I unplugged the HDMI cable order was restored.

 free -h
total used free shared buff/cache available
Mem: 927M 81M 629M 7.6M 215M 787M
Swap: 349M 0B 349M

I’m not sure I should be proud I fixed the issue or embarrassed it took me this long to unplug the monitor. Either way, I definitely won’t be making the same mistake again.