Conversation
Removes lokimq::string_view (the type alias is still provided for backwards compat, but now is always std::string_view). Bump version (on dev branch) to 1.2.0
In C++17 all the recv calls have a [[nodiscard]], so check them as part of the test (both to silence the warnings and for better test code).
Make the char handling a bit more generic so that std::byte (or other size-1 types) will work.
Previously you could only generate a string from a string_view, or could manage the string yourself and pass input iterators plus an output iterator. This commit adds an intermediate version that creates a string from a pair of input iterators.
Various small C++17 code improvements. Replace mapbox::variant with std::variant. Remove the bt_u64 type wrapper; instead we know have `bt_value` which wraps a variant holding both int64_t and uint64_t, and has contructors to send signed/unsigned integer types into the appropriate one. lokimq::get_int checks both as appropriate during extraction. As a side effect this means we no longer do the uint64_t -> int64_t conversion on the wire, ever, without needing the wrapper; although this can break older versions sending large positive integers (i.e. larger than int64_t max) those weren't actually working completely reliably with mapbox variant anyway, and the one place using such a value in loki core (in a checksum) is already fully upgraded across the network (currently using bt_u64, but always sending a positive value on the wire).
These (...) lambdas are va_arg which cause corruption under clang on both linux and macos. They weren't supposed to be va_args -- switch them to the intended universal references, which fixes the crash.
We call libsodium functions which require a sodium_init() call; this is usually a no-op (zmq will have already called it for us), but in case zmq is built with tweetnacl instead of sodium we need to call it before we call it directly in the LokiMQ ctor and the test suite.
The test suite needs this, in particular.
This was breaking if we didn't find libzmq (or didn't find recent enough) because the target didn't exist.
Adds CI builds (using drone) for various ubuntu/debian builds on amd64/arm64/i386/armhf.
macos sometimes needs more time here
This adds to ability to have lokimq manage specific threads to which jobs (individual, batch jobs, batch completions, or timers) can be directed to. This allows dedicating a thread to some slow or thread-unsafe action where you can dump jobs to the tagged thread as a method of lockless job queuing.
This allows use of some free functions within lokimq that can still log (assuming they have a LokiMQ reference).
This renames the class to make it clearer what it does, and drops the .name attribute from it so that it can cheaply be passed around. This then means it can be cheaply passed by value (using std::optionals) rather than by pointer when specifying a thread.
The init function doesn't seem all that useful and makes the interface a bit more complicated, so drop it. Also addresses a race condition that can happen with tagged thread startup when the proxy tries to talk to a tagged thread but the tagged thread hasn't connected yet (which then aborts the proxy because it assumes workers are always routable).
There can be a spurious failure here if the backdoor_details element
hasn't been added yet, so lock & check it when waiting for the test
conditions.
The weirdest thing about this error is that it can fail but then when
expanding values they expand to *correct* values, i.e. so you get:
FAILED:
REQUIRE( backdoor_details == all_the_things )
with expansion:
{ "Alaska", "I'm the luckiest man in the world", "Loretta", "because all my
life are belong to Google", "moustache hatred", "photos", "scallops",
"snorted when she laughed", "tickled pink" }
==
{ "Alaska", "I'm the luckiest man in the world", "Loretta", "because all my
life are belong to Google", "moustache hatred", "photos", "scallops",
"snorted when she laughed", "tickled pink" }
init() got removed during the tagged threads PR, but the documentation didn't get updated. Fixed it.
If the LokiMQ object gets destroyed before having called `start()` then we'd end up destroying the threads for tagged workers without joining them. This listens on the internal worker socket (normally the domain of the proxy thread) and tells them to QUIT if such a destruction happens.
The `join()`s could hang if the tagged worker threads aren't ready yet, so retry sending until they become routable.
data_parts() wasn't currently used anywhere, and was broken: it was calling bt_deserialize which was just wrong. This repurposes it to take iterators over strings (or string-like types) and append those parts as message parts. Also adds tests for it.
On the wire they are just lists, but this lets you put tuples onto and pull tuples off of the wire. (Also supports std::pair). Supports direct serialization (via bt_serialize()/bt_deserialize()), list/dict consumer deserialization, and conversion from a bt_value or bt_list via a new bt_tuple() function.
Makes the TaggedThreadID copyable.
…D similar to how STARTUP_UMASK is done.
Add var::get/var::visit implementations of std::get/std::visit that get used if compiling for an old macos target, and use those. The issue is that on a <10.14 macos target Apple's libc++ is missing std::bad_variant_access, and so any method that can throw it (such as std::get and std::visit) can't be used. This workaround is ugly, but such is life when you want to support running on Apple platforms.
Apple, in particular, often fails tests with an address already in use if attempt to reuse a port that the process just closed, because it is a wonderful OS.
Updates bundled cppzmq to 4.7.1, and replaces deprecated functions with new API.
The thread_local `std::map` here can end up being destructed *before* the LokiMQ instance (if both are being destroyed during thread joining), in which case we segfault by trying to use the map. Move the owning container into the LokiMQ instead (indexed by the thread) to prevent that. Also cleans this code up by: - Don't close control sockets from the proxy thread; socket_t's aren't necessarily thread safe so this could be causing issues where we trouble double-closing or using a closed socket. - We can just let them get closed during destruction of the LokiMQ. - Avoid needing shared_ptr's; instead we can just use a unique pointer with raw pointers in the thread_local cache. This simplifies closing because all closing will happen during the LokiMQ destruction.
This is making lokimq headers & static lib get installed when lokimq is used as a project subdirectory, which is very annoying. This adds an option for enabling the install lines, and only enables it if doing a shared library or a top-level project build.
`is_hex()` is a bit misleading as `from_hex()` requires an even-length
hex string, but `is_hex()` also allows odd-length hex strings, which
means currently callers should be doing `if (lokimq::is_hex(str) &&
str.size() % 2 == 0)`, but probably aren't.
Since the main point of `lokimq/hex.h` is for byte<->hex conversions it
doesn't make much sense to allow `is_hex()` to return true for something
that can't be validly decoded via `from_hex()`, thus this PR changes it
to return false.
If someone *really* wants to test for an odd-length hex string (though
I'm skeptical that there is a need for this), this also exposes
`is_hex_digit` so that they could use:
bool all_hex = std::all_of(str.begin(), str.end(), lokimq::is_hex_digit<char>)
Make lokimq::is_hex check for size being a multiple of 2
Decoding into a std::byte output iterator was not working because the `*out++ = val` assignment doesn't work when the output is std::byte and val is a char/unsigned char/uint8_t. Instead we need to explicitly cast, but figuring out what we have to cast to is a little bit tricky. This PR makes it work (and bumps the version for this and the is_hex fix).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.