Bug#118927 Fix incorrect hex representation in SHOW CREATE LIBRARY #628
Conversation
This commit fixes the incorrect hexadecimal representation of binary
libraries in SHOW CREATE LIBRARY output. The issue produces malformed
hex literals where some bytes are incorrectly expanded with "FFFFFF"
patterns, violating the standard convention that each byte should be
represented by exactly 2 hex digits.
For example, a binary library would incorrectly display as:
0x3872FFFFFFD3FFFFFFBBFFFFFFA6...
Instead of the correct standard representation:
0x3872D3BBA66A27A9...
This incorrect representation produces bloated output that violates
universal hexadecimal formatting conventions. On x86_64 (where char
is signed by default), negative byte values undergo sign extension,
resulting in "FFFFFF" patterns in the hex representation. On AArch64
(where char is unsigned by default), these patterns are absent.
The fix:
- Modifies the binary_to_hex function to mask each byte value with & 0xFF,
ensuring each byte is properly represented with exactly 2 hex digits
- Updates the sp-library-binary.result test file to reflect the corrected
hex representation
This contribution is under the OCA signed by Amazon and covering
submissions to the MySQL project.
|
Hi, thank you for submitting this pull request. In order to consider your code we need you to sign the Oracle Contribution Agreement (OCA). Please review the details and follow the instructions at https://oca.opensource.oracle.com/ |
|
This contribution is under the OCA signed by Amazon and covering submissions to the MySQL project. |
|
Hi @akshatnehra, Thanks for contributing to MySQL |
|
Hi, thank you for your contribution. Please confirm this code is submitted under the terms of the OCA (Oracle's Contribution Agreement) you have previously signed by cutting and pasting the following text as a comment: |
|
I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it. |
|
Hi, thank you for your contribution. Your code has been assigned to an internal queue. Please follow |
Description
The SHOW CREATE LIBRARY command produces an incorrect hexadecimal representation of binary libraries on x86_64 architectures. On x86_64 platforms (where char is signed by default), the command inserts erroneous "FFFFFF" patterns before each byte with the high bit set, resulting in an invalid and bloated hex representation.
For example, on x86_64:
AS
0x3872FFFFFFD3FFFFFFBBFFFFFFA66A27FFFFFFA9FFFFFFA34F3834FFFFFFF3FFFFFF82FFFFFFBC1A7360FFFFFFD3FFFFFFBA0C5A7761FFFFFFE5FFFFFFE67A7BWhile on AArch64:
AS
0x3872D3BBA66A27A9A34F3834F382BC1A7360D3BA0C5A7761E5E67A7BThe AArch64 representation is actually correct, as each byte is properly represented by exactly 2 hex digits, which is the standard way to represent binary data in hexadecimal format. On x86_64 (where char is signed by default), negative byte values undergo sign extension, resulting in "FFFFFF" patterns in the hex representation. On AArch64 (where char is unsigned by default), these patterns are absent.
This inconsistency causes test failures across different architectures and produces misleading, non-standard hexadecimal representations.
How to recreate the issue
Run the MySQL Test Run (MTR) test main.sp-library-binary on both x86_64 and AArch64 architectures:
The test fails on AArch64 because it expects the incorrect x86_64 representation with "FFFFFF" patterns, but AArch64 correctly produces a standard hex representation.
Fix
Prevent sign extension on all platforms by using a bit mask to isolate only the relevant 8 bits of each byte:
This approach masks off the higher bits with & 0xFF, preventing sign extension regardless of platform. It would produce the more compact and correct representation (matching current AArch64 behavior) on all architectures.
Update
sp-library-binary.resultfile to match the correct behavior.Testing
I have tested this fix on both x86_64 and AArch64 platforms and with the fix applied, both platforms produce identical, correct output.
This contribution is under the OCA signed by Amazon and covering submissions to the MySQL project.