Under-voltage messages in the kernel log?

[notro]

I've been reminded that inadequate power supplies continous to be a problem for Raspberry Pi users (and even seasoned ones can be bitten by it).
This is especially true for those running headless as there is no blinking under-voltage icon on the screen to get your attention.

I've had this in the back of my mind for a couple of years that it would be nice to have a message in the kernel log when under-voltage occurs. So maybe now is the time to see if I can do something about this.

The first question is ofc, would this be helpful for more people than just myself?

The next question is how to get to this info from the kernel. Is there a mailbox property for this?
Either way I guess polling is the only option here.
On the Pi2 I believe it's possible to use the pwr led gpio as an interrupt to catch events, but on the Pi3 the pwr led 'gpio chip' probably doesn't have interrupt capability that can be easily routed to the ARM.

It would also be nice to have a message in the log if the previous boot/run had an under-voltage event.
So if the watchdog rescues an under-voltage induced crash, under-voltage as one possible reason will be shown in the log after reset.

[popcornmix]

vcgencmd get_throttled has the information needed by the arm, but the gencmd/vchiq interface is quite heavyweight for a kernel driver. I could add a mailbox command to get the same information.

What sort of driver are you thinking of that could do the polling?

How are you thinking of storing the fact that previous run had an under-voltage condition?

The reset controller (RSTS) is the only state that survives a reboot, but we are already using the bits available there. sdcard is the other option, but we may not run long enough after the event to get the data safely onto the sdcard.

[notro]

What sort of driver are you thinking of that could do the polling?

I was hoping that the power_supply class could be useful for this, but it seems to be more or less dedicated to charging and maybe not a good candidate for detecting under-voltage.

Another possibility is to do a regulator driver for the 5v0 regulator. Regulators do have an event for under-voltage at least: REGULATOR_EVENT_UNDER_VOLTAGE.

We can't just write a simple module that does this since we need a DT device to get to the firmware device for querying properties.

How are you thinking of storing the fact that previous run had an under-voltage condition?

I suspected thet the RSTS was used up already so I hoped that maybe the firmware had access to some other register that survives reboot.

It is possible to preserve panic messages across reset as I found out: Using ramoops/pstore to capture kernel panics

Sooo, maybe this just as well can be written as a systemd service that polls the value and writes events to the kernel log, and then use ramoops to preserve the event across reboot.

Do you think under-voltage logging is useful and worth spending time on for the community?

[popcornmix]

Do you think under-voltage logging is useful and worth spending time on for the community?

Yes.

[FishTest]

In most cases, the raspberry pi runs as a server and does not have a screen. There is no proprietary PMIC in the raspberry pi, and the Linux power supply driver can not communicate with the PMIC. So it's very important to get some power information in other ways.

[popcornmix]

I've added a commit to firmware that supports tag GET_THROTTLED = 0x00030046

/opt/vc/bin/vcmailbox 0x00030046 4 4 0
0x0000001c 0x80000000 0x00030046 0x00000004 0x80000004 0x00060006 0x00000000

The 0x00060006 is the throttled status as described here.
Will be in next firmware build (possibly tomorrow).

[notro]

I've been thinking about the challenge of polling and reliably getting info about anything but the first under-voltage event. We would have to poll very fast to be certain that we can detect an ongoing under-voltage.

How fast would we have to poll to be certain to detect under-voltage?

One solution is to have an under-voltage bit that is reset when read.
This way the polling period doesn't matter (that much), we are able to say that an event happened some time during the last n seconds.

[popcornmix]

I believe this is already handled. We added this to vcgencmd get_throttled for bcmstat.

Basically the bottom 16 bits are the live values and top 16 bits are the sticky values.
If you pass in a parameter to get_throttled (either gencmd or mailbox) it will clear the sticky bits after returning them.

There is a slight caveat if multiple users are clearing bits, but the result isn't fatal and if a kernel driver is supporting this functionality then we can deprecate other users.

[notro]

Indeed it does, I didn't come across that functionality in my web searches.

pi@pib:~ $ vcgencmd get_throttled 0x05
throttled=0x50000
pi@pib:~ $ vcgencmd get_throttled 0x05
throttled=0x0

[popcornmix]

Latest rpi-update kernel includes the GET_THROTTLED mailbox message.

[notro]

Even if I ask to clear the sticky bit, it is stuck for ~5 seconds:

$ while true; do vcgencmd get_throttled 1; sleep 1; done
throttled=0x40000
throttled=0x40000
throttled=0x40000
<brief overload>
throttled=0x50005
throttled=0x50005
throttled=0x50005
throttled=0x50000
throttled=0x40000
throttled=0x40000
throttled=0x40000
throttled=0x40000
^C

Is it a feature or a bug?

[notro]

I can add that I'm looking at 3 use cases:

1. Logging to /var/log/get_throttled.log and /dev/kmsg, runs as root. Needs to clear the sticky bit in order to track changes.

2. Desktop notification, must run as user. Not sure how to do this since the root process is clearing the sticky bit(s). It is possible for the process running as root to send events via the dbus system bus, but I'm afraid that it will be complicated, and I want this to be as simple as possible.

3. Realtime monitoring hopefully with multiload-ng which can graph output from commands. This one polls maybe once a second so it can look at the current value.

[popcornmix]

The "live" values for get_throttled have a decay so you can catch brief events (corresponds with the length of time the warning symbol appears on display).
Under-voltage is implemented as a uint32_t history that shifts and ors each cycle (100ms). So just over 3 seconds of decay.

[popcornmix]

Can the kernel driver be the master who reads and clears sticky bits and all other users go through the kernel driver (e.g. kernel exports values to /sys or similar). That avoids other users clearing the sticky bits.