Fuzzing The Gleam Compiler | kurz.net

Can you find bugs in a compiler by generating random programs? Published on: Tue Aug 25 2026 Introduction I regularly check on Gleam’s changelog and issue tracker. I am very fond of this project and the people contributing to it. But every time I see an issue that relates to code generation or different outputs between the Erlang and JavaScript, it nags me that there was no way to basically “compute all the Gleam programs”, run them and see if there are any issues. I imagine it like a chessboard, where you have a quasi-infinite number of possible positions on the board. But we want the chessboard to contain Gleam programs and we want an infinitely big database of those programs to see if they uncover untested edge cases. My first attempt of doing something adjacent to this was actually prompting an LLM. I instructed it to read through loads of past Gleam issues and find more edge cases by “thinking hard about it”. It came up with all sorts of bit array combinations, nested anonymous functions, nested use patterns. Predictably, this approach did not yield many results. $20 bucks of tokens later, it found exactly one issue, which was reported and fixed right away: https://github.com/gleam-lang/gleam/issues/5613. One is definitely more than zero. But there are plenty of issues with “LLM fuzzing”: it’s pricey, not deterministic and a bit like pulling the lever on a slot machine. But there was another idea that I had avoided pursuing, because to be honest it just sounded like a lot of work: structure-aware fuzzing. Structure-aware Fuzzing Writing software is hard, and humans are not great at it. To help, we’ve built other software that can partially automate the search for bugs. One of these programs is a fuzzer. They generate randomized inputs to feed into our program. The premise is that on a large scale, these random inputs will distribute in such a way that edge cases will be surfaced that we haven’t thought of yet. Fuzzers can range from totally random scrambled bytes, to highly structured grammar-aware ASTs. Feeding totally random bytes to a program is usually done for use cases that are working with images, files, network requests, protocols, etc. There are plenty of examples where fuzzing found real security flaws and bugs in open source software. For example, this finding by zzuf in Firefox, where flipping some bits in an image file would result in a browser crash: https://nvd.nist.gov/vuln/detail/CVE-2007-6715. But fuzzers have also uncovered real exploitable security flaws via buffer overflows. There is a program by Google “OSS Fuzz” that continuously fuzzes a lot of important open source projects: https://google.github.io/oss-fuzz/ In our case, we are not working on a browser or network protocol. We have a compiler. And that opens up the possibility for structure-aware fuzzing. That means that we do not generate a stream of random bytes, but rather a stream of code in the form of source code or an AST. Enter Gleam There are a few things about Gleam that make it a particularly interesting candidate for fuzzing. It generates code for two targets: JavaScript and Erlang. We can compare the output of the same program for both targets and flag any differences. Gleam has a minimalistic syntax. At least compared to most other popular programming languages. We can generate valid programs that cover almost all concepts provided by the language with relatively little code. Static types. Needless to say, this is an amazing feature that lets us ensure that a program will not crash at runtime. That doesn’t mean there can’t be any bugs in the type system. There have been issues related to type inference in the past. But as we will learn later on, each aspect of the language will require its own testing approach. The functional nature and the fact that everything is an expression makes composing and structuring the programs very convenient. Rust. This might be easy to overlook, but the fact that the Gleam compiler itself is written in Rust makes it very easy to integrate existing fuzzing tooling. We can test parts of the compiler without having to run a single .gleam file. Resources I Used We are going to dive into more technical aspects of the fuzzer. But I am not going to go into a lot of code or detail. If you would like to read more about that, do check out this post and blog by Nick Fitzgerald. It served as the main inspiration for this project: https://fitzgen.com/2020/08/24/writing-a-test-case-generator.html Our fuzzer is going to be generation-based, not mutation-based. If you would like to understand the difference better, I recommend reading this article: https://fitzgen.com/2026/06/01/structure-aware-fuzzing-experiment.html In the article, the author comes to the conclusion that, at least for wasm, the mutation-based approach found a lot more issues than the generation-based approach. So it is probably worth implementing for this project in the future! For an even more in-depth dive into the topic, check out this resource: https://www.fuzzingbook.org/. You can find the full code for the Gleam fuzzer in this branch of my Gleam fork: https://github.com/daniellionel01/gleam/tree/fuzzing Phase 1: The Parser An important design choice for the fuzzer: use the public compiler API. Even though there might not be any stability guarantees for the compiler API, this makes it easy to stay compatible with future versions of Gleam. It also avoids fiddling with implementation details, which is a good way to ensure we’re not creating any false positives or negatives. To see some examples of how our parser catches and categorizes the outputs: $ cargo run -p fuzzing-core --example classify "pub fn main() { 1 }" -> compiled (js: 39B, ts: 32B, erl: 238B) "pub fn main() { let f = fn(x) { x + 1 }; f(41) }" -> parse error "pub fn main() { 1 +. \"x\" }" -> analysis rejected (javascript) "pub fn main() {" -> parse error Using the fuzz crate and some wrapper code, we can very quickly spam the Gleam compiler with randomly generated inputs (not structured yet), to see if we can crash the compiler instead of giving us an error message with more context. We’re only going to run it for 1 second, because the output is quite large: $ cargo +nightly fuzz run parse_only --fuzz-dir fuzzing-harness -- -max_total_time=1 -timeout=10 INFO: Running with entropic power schedule (0xFF, 100). INFO: Seed: 302379076 INFO: Loaded 1 modules (740945 inline 8-bit counters): 740945 [0x105eeac70, 0x105f9fac1), INFO: Loaded 1 PC tables (740945 PCs): 740945 [0x105f9fac8,0x106aedfd8), INFO: 2466 files found in fuzzing-harness/corpus/parse_only INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes INFO: seed corpus: files: 2466 min: 1b max: 4046b total: 425422b rss: 62Mb #2467 INITED cov: 2434 ft: 8563 corp: 1249/171Kb exec/s: 0 rss: 108Mb #2513 REDUCE cov: 2434 ft: 8563 corp: 1249/171Kb lim: 3764 exec/s: 0 rss: 108Mb L: 48/3753 MS: 1 EraseBytes- #2645 REDUCE cov: 2434 ft: 8563 corp: 1249/171Kb lim: 3764 exec/s: 0 rss: 108Mb L: 8/3753 MS: 2 ChangeBit-EraseBytes- #2656 REDUCE cov: 2434 ft: 8563 corp: 1249/171Kb lim: 3764 exec/s: 0 rss: 109Mb L: 2/3753 MS: 1 EraseBytes- #2937 REDUCE cov: 2434 ft: 8563 corp: 1249/171Kb lim: 3764 exec/s: 0 rss: 109Mb L: 314/3753 MS: 1 EraseBytes- #3183 NEW cov: 2434 ft: 8578 corp: 1250/172Kb lim: 3764 exec/s: 0 rss: 110Mb L: 1054/3753 MS: 1 CopyPart- #3591 REDUCE cov: 2434 ft: 8578 corp: 1250/172Kb lim: 3764 exec/s: 0 rss: 111Mb L: 99/3753 MS: 3 ShuffleBytes-CrossOver-EraseBytes- #3934 NEW cov: 2434 ft: 8585 corp: 1251/173Kb lim: 3764 exec/s: 0 rss: 112Mb L: 399/3753 MS: 3 CMP-CopyPart-CopyPart- DE: "\010\000\000\000\000\000\000\000"- # ... NEW_FUNC[1/7]: 0x0001031cfb88 in _RINvNtCs3kGMwX4aip8_4core3ptr9drop_glueINtNtCshX1O598ANu2_5alloc3vec3VecINtNtNtCs846PmCUGaYz_10gleam_core3ast8constant8ConstantuEEEB1f_+0x0 (parse_only:arm64+0x1002abb88) NEW_FUNC[2/7]: 0x00010323bff4 in _RINvNtCs3kGMwX4aip8_4core3ptr9drop_glueINtNtNtCs846PmCUGaYz_10gleam_core3ast8constant8ConstantuEEBI_+0x0 (parse_only:arm64+0x100317ff4) #16785 NEW cov: 2483 ft: 8694 corp: 1262/177Kb lim: 3786 exec/s: 16785 rss: 144Mb L: 85/3753 MS: 1 CrossOver- #18061 REDUCE cov: 2483 ft: 8694 corp: 1262/177Kb lim: 3797 exec/s: 18061 rss: 149Mb L: 403/3753 MS: 1 EraseBytes- NEW_FUNC[1/4]: 0x00010312c3b0 in _RINvMs_NtCs846PmCUGaYz_10gleam_core5parseINtB5_6ParserINtNtB5_5lexer5LexerINtBT_14NewlineHandlerINtNtNtNtCs3kGMwX4aip8_4core4iter8adapters3map3MapNtNtNtB1F_3str4iter11CharIndicesNCNvBT_14make_tokenizer0EEEE23parse_bit_array_segmentNtNtNtB7_3ast7untyped11UntypedExprNCNCNvB2_21parse_expression_units6_00NvB2_17expect_expressionNvB5_24bit_array_expression_intEB7_+0x0 (parse_only:arm64+0x1002083b0) NEW_FUNC[2/4]: 0x00010318893c in _RINvNtCs3kGMwX4aip8_4core3ptr9drop_glueINtNtCs846PmCUGaYz_10gleam_core3ast15BitArraySegmentNtNtBE_7untyped11UntypedExpruEEBG_+0x0 (parse_only:arm64+0x10026493c) # ... ###### Recommended dictionary. ###### "\010\000\000\000\000\000\000\000" # Uses: 879 "\201\000" # Uses: 941 ###### End of recommended dictionary. ###### Done 24887 runs in 2 second(s) Sweet. Looking at some of the artifacts it produces, you can see what kind of inputs are generated: fn ar(n,n,n,///A# o ఌఌ「彸䕅䕅ⅅ䕅+ ">\u{000000000000000000.%\f0 fn ar(n ar:rn a( The nice thing about this is that it can test everything without running the gleam binary at all. It runs in-memory with the compiler pipeline in Rust. And guess what! When I let this fuzzer run for quite a while, it actually found a regression on nightly, which did not happen on v1.18.1 (which was the latest version of Gleam at the time of writing this): $ cargo +nightly fuzz run --fuzz-dir fuzzing-harness parse_only fuzzing-harness/artifacts/parse_only/crash-8b14db5e4bf152924501e0818787026e9f5ea229 =fuzzing-harness/artifacts/parse_only/ fuzzing-harness/artifacts/parse_only/crash-8b14db5e4bf152924501e0818787026e9f5ea229` INFO: Running with entropic power schedule (0xFF, 100).