fix(metal): filter compiler errors from JIT header dependency parsing #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The make_compiled_preamble.sh script uses the Metal compiler's -H flag to determine header dependencies for JIT source generation. On macOS 15 SDK, when MetalPerformancePrimitives headers are not available, the compiler outputs error messages to stderr which were incorrectly parsed as header paths. This caused critical headers like params.h (containing GEMMParams struct) to be omitted from the generated JIT source, resulting in runtime Metal compilation errors: "unknown type name 'GEMMParams'" The fix filters the -H output to only keep lines matching the expected format: one or more dots followed by an absolute path. This cleanly removes error messages, warnings, and other unexpected compiler output.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem: Metal Compiler Error Messages Corrupt JIT Header Parsing
Unfiltered Output (Before Fix)
When the Metal compiler encounters missing headers, error messages are mixed with valid -H output:
. /path/to/mlx/backend/metal/kernels/steel/gemm/params.h
/tmp/test_error.h:2:10: fatal error: 'nonexistent_header.h' file not found
#include "nonexistent_header.h"
^~~~~~~~~~~~~~~~~~~~~~
. /path/to/mlx/backend/metal/kernels/steel/gemm/gemm_nax.h
.. /path/to/mlx/backend/metal/kernels/steel/gemm/nax.h
... /path/to/mlx/backend/metal/kernels/steel/defines.h
... /path/to/mlx/backend/metal/kernels/steel/gemm/transforms.h
.... /path/to/mlx/backend/metal/kernels/steel/utils.h
... /path/to/mlx/backend/metal/kernels/steel/utils/integral_constant.h
.... /path/to/mlx/backend/metal/kernels/steel/utils/type_traits.h
... /path/to/Xcode.app/.../MetalPerformancePrimitives.framework/Headers/MetalPerformancePrimitives.h
.... /path/to/Xcode.app/.../MetalPerformancePrimitives.framework/Headers/MPPTensorOpsConvolution2d.h
.... /path/to/Xcode.app/.../MetalPerformancePrimitives.framework/Headers/MPPTensorOpsMatMul2d.h
1 error generated.
The script parses this by splitting on whitespace and pairing depth indicators (dots) with paths. Error messages break this pairing, causing headers like params.h to be lost.
Filtered Output (After Fix)
With the fix (grep -E "^.+ /"), only valid header lines are kept:
. /path/to/mlx/backend/metal/kernels/steel/gemm/params.h
. /path/to/mlx/backend/metal/kernels/steel/gemm/gemm_nax.h
.. /path/to/mlx/backend/metal/kernels/steel/gemm/nax.h
... /path/to/mlx/backend/metal/kernels/steel/defines.h
... /path/to/mlx/backend/metal/kernels/steel/gemm/transforms.h
.... /path/to/mlx/backend/metal/kernels/steel/utils.h
... /path/to/mlx/backend/metal/kernels/steel/utils/integral_constant.h
.... /path/to/mlx/backend/metal/kernels/steel/utils/type_traits.h
Result