How can I catch hung or stuck I/Os with FIO?

[aqibahmad]

Does FIO has any way currently to report I/Os that have been stuck for a long time (time after which to report may be specified using a parameter)?

I raised this RFE but wanted to check if there is any existing or alternate way of doing this. #2012

[sitsofe]

Hi @aqibahmad:

In short no, fio does not currently have a way to report I/Os that never return. The closest you get is max_latency which aborts the whole job after an I/O that has taken too long returns.

If you've ever done I/O to a file on NFS on *nix systems and then had the NFS server go away you will know that certain configurations can mean the process becomes stuck in uninterruptible sleep and is essentially unkillable, so any synchronous ioengine doing I/O through regular synchronous I/O syscalls could face this issue. You could have some sort of alarm thread/signal that let you know you had gone over time but at that point what do you do? I guess you could abort and give up but even then you wouldn't be able to shut fio down fully until the I/O finally returned...

The situation is a bit better for an asynchronous I/O engine (at least fio isn't stuck making a call) but you would need to rework fio to constantly check if in-flight I/Os when waiting for them to complete. The next problem is what do you do if you do find such an I/O? If you want to do anything other than aborting, the underlying mechanism needs to support I/O cancellation otherwise you can end up in a situation where you've given up but then the I/O finally finishes etc...

If the mechanism used by the ioengine itself supports timing out on commands then life is simpler as you "just" set a timeout on that that, causing the I/O to return failure if it takes too long (back when I did iSCSI work such a thing could be set on the initiator and multipath daemon) but whether this is possible is a case-by-case thing.

So in some cases you might be able to alter a specific I/O engine to set a timeout, you might be able to something for async ioengines (but it's going to be messy) and dealing with a synchronous ioengine that can hang looks tough. However I'm not aware of anyone even planning to take on this work.