Event camera simulation

[saihv]

This PR introduces a simple event camera simulation in Python, using numba for performance.

The event simulator uses two consecutive RGB images (converted to grayscale), and computes "events" based on the change in log luminance between the images. These events are reported as a stream of bytes, following this format:

<x> <y> <timestamp> <pol>

x and y are the pixel locations of the event firing, timestamp is the global timestamp in microseconds and pol is either +1/-1 depending on whether the brightness increased or decreased. Along with this bytestream, an accumulation of events over a 2D frame is also constructed, known as an 'event image' that visualizes +1 events as red and -1 as blue pixels.

There are quite a few parameters that can be tuned to achieve a level of visual fidelity/performance. The main factors would be the resolution of the camera and the log luminance threshold (TOL) that determines whether or not a detected change counts as an event. There is also currently a max limit on the number of events generated per pair of images, which can also be tuned.

• Event simulator backend
• Event image visualization
• Return list of bytes for each new image obtained
• Save events to a file if needed

[zimmy87]

Most comments are style/debugging-related. Currently blocked from testing due to the last comment though.

[zimmy87]

looks good to me

[sumzora]

Hello, I want to use event camera in Airsim. when I run <test_event_sim.py> , the error comes as followes and the <events.pkl> is blank. (Python = 3.9 AirSim = 1.5 UE=4.26)
ERRORS
Initialized event camera simulator with sensor size: (64, 64)
D:\RL\AirSimv1.5\PythonClient\eventcamera_sim\test_event_sim.py:91: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
np.fromstring(response[0].image_data_uint8, dtype=np.uint8), event_generator.rgb_image_shape,)
D:\RL\ANACONDA\envs\ASclient\lib\site-packages\numba\core\typed_passes.py:329: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.

File "event_simulator.py", line 24:
@njit(parallel=True)
def esim(
^

warnings.warn(errors.NumbaPerformanceWarning(msg,
D:\RL\AirSimv1.5\PythonClient\eventcamera_sim\test_event_sim.py:91: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
np.fromstring(response[0].image_data_uint8, dtype=np.uint8), event_generator.rgb_image_shape,)

[JamesYang110043]

Hello, I want to use event camera in Airsim. when I run <test_event_sim.py> , the error comes as followes and the <events.pkl> is blank. (Python = 3.9 AirSim = 1.5 UE=4.26) ERRORS Initialized event camera simulator with sensor size: (64, 64) D:\RL\AirSimv1.5\PythonClient\eventcamera_sim\test_event_sim.py:91: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead np.fromstring(response[0].image_data_uint8, dtype=np.uint8), event_generator.rgb_image_shape,) D:\RL\ANACONDA\envs\ASclient\lib\site-packages\numba\core\typed_passes.py:329: NumbaPerformanceWarning: The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.

File "event_simulator.py", line 24: @njit(parallel=True) def esim( ^

warnings.warn(errors.NumbaPerformanceWarning(msg, D:\RL\AirSimv1.5\PythonClient\eventcamera_sim\test_event_sim.py:91: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead np.fromstring(response[0].image_data_uint8, dtype=np.uint8), event_generator.rgb_image_shape,)

I have a same problem here, and I got the same error log.
Maybe "Namba" lead to this error. But I don't have any idea.
Did you solved it? Thx