Add Snapshotter

[vdusek]

Description

• add Snapshotter component;
• adjust SystemStatus, LocalEventManager, and relevant data classes;
• some functionality for system measurement was moved to crawlee/_utils/system.py;
• add unit tests to cover new code

Additional notes

• Removing crawlee.config module for now since we do not need it. Let's add it again in the future and move some fields to it if needed.
• Regarding the type casting (e.g. cast(list[Snapshot], self._memory_snapshots)) in Snapshotter. It is there because of this https://mypy.readthedocs.io/en/stable/common_issues.html#variance, I am not sure if we can address it better.

Testing

Unit tests

• Unit tests should cover new code.

---------- coverage: platform linux, python 3.9.18-final-0 -----------
Name                                         Stmts   Miss  Cover
----------------------------------------------------------------
...
src/crawlee/autoscaling/snapshotter.py         112      2    98%
src/crawlee/autoscaling/system_status.py        55      4    93%
...

Manual testing / execution

• For manual testing or just execution of the new components, check the following script.

from __future__ import annotations

import asyncio
from datetime import timedelta

from crawlee.autoscaling import Snapshotter, SystemStatus
from crawlee.events import LocalEventManager


async def test_snapshotter() -> None:
    async with LocalEventManager(system_info_interval=timedelta(seconds=1)) as event_manager:
        snapshotter = Snapshotter(
            event_manager=event_manager,
            snapshot_history=timedelta(seconds=5),
            max_event_loop_delay=timedelta(milliseconds=1),  # to simulate overloading
        )

        await snapshotter.start()

        for _ in range(10):
            print('waiting...\n')
            await asyncio.sleep(1)

            cpu_sample = snapshotter.get_cpu_sample()
            print(f'cpu_sample: {cpu_sample}\n')

            mem_sample = snapshotter.get_memory_sample()
            print(f'mem_sample: {mem_sample}\n')

            event_loop_sample = snapshotter.get_event_loop_sample()
            print(f'event_loop_sample: {event_loop_sample}\n')

        system_status = SystemStatus(snapshotter)
        current_status = system_status.get_current_status()
        print(f'status: {current_status}\n')

        historical_status = system_status.get_historical_status()
        print(f'hstatus: {historical_status}')

        await snapshotter.stop()


if __name__ == '__main__':
    asyncio.run(test_snapshotter())

[janbuchar]

I'm kinda sad from seeing all those protected member accesses. Is there any way we could refactor the snapshotter to avoid this? Is this test really valuable? From what I see here, we're really just verifying some internal details, not the behavior of the Snapshotter class.

[vdusek]

How would you test otherwise that the snapshotter correctly read and stored the used ratio of the CPU? I don't know, it seems valuable to me to have these checks there.

[janbuchar]

I believe that it would be better to set up an EventManager and have it emit a SystemInfo event. And then, instead of looking through the snapshots property, to use one of those get_*_system_info methods.

[vdusek]

Yeah, but we would need to either directly invoke the private method _emit_system_info_event or wait for the system_info_interval to trigger its execution by RecurringTask.

And we would have to update it for all other test_snapshot_*.

Also, isn't this a better unit test approach? I mean, to have it isolated. When a bug in the RT occurs we would see that only unit tests for RT are failing and will be able to easily identify where the problem is. Rather than failing everything and looking for the cause.

I understand your point of touching just the public interface in the tests, but I'd prefer to stay with the current implementation in this case.

[janbuchar]

Yeah, but we would need to either directly invoke the private method _emit_system_info_event or wait for the system_info_interval to trigger its execution by RecurringTask.

Or we could make a testing implementation of EventManager where emitting events could be done from the outside (I mean from the test).

And we would have to update it for all other test_snapshot_*.

Yes, but I'd consider that a benefit 🙂

Also, isn't this a better unit test approach? I mean, to have it isolated. When a bug in the RT occurs we would see that only unit tests for RT are failing and will be able to easily identify where the problem is. Rather than failing everything and looking for the cause.

Well, depends what you consider to be an unit, but for me, this is a sub-unit test. You're right that using a non-trivial event manager would make it closer to an integration test. With a fake (not sure if that's correct TDD terminology) implementation, I don't see an issue.

[vdusek]

Ok, agreed. Taking into account the current situation, I opened a new issue for that - #73. Let's improve the testing of the Snapshotter later and merge this now.