Reproducibility: Track external commands used

[osresearch]

Running strace -f -e execve make allows us to see what external programs are used and how many times in the build process. If we want to ensure that only trusted/reproducible tools are used, then we might want to build or depend on our own versions of these.

[osresearch]

Excluding the musl-cross build, a normal build executes these programs this many times:

  19091 /usr/bin/sed
  13614 /usr/bin/rm
  12855 /root/heads/crossgcc/bin/x86_64-linux-musl-gcc
  12300 /root/heads/crossgcc/bin/../libexec/gcc/x86_64-linux-musl/8.3.0/cc1
  11125 /bin/sh
   9889 /usr/bin/cat
   7518 /bin/bash
   7093 /root/heads/crossgcc/bin/../lib/gcc/x86_64-linux-musl/8.3.0/../../../../x86_64-linux-musl/bin/as
   5953 /usr/bin/mv
   3872 /usr/bin/tail
   2542 scripts/basic/fixdep
   2397 /usr/bin/mkdir
   2381 /usr/bin/grep
   1926 ./tools/objtool/objtool
   1409 /usr/bin/expr
   1373 /usr/bin/dirname
   1251 /usr/bin/tr
   1201 /root/heads/crossgcc/bin/../lib/gcc/x86_64-linux-musl/8.3.0/../../../../x86_64-linux-musl/bin/ld
   1200 /root/heads/crossgcc/bin/../libexec/gcc/x86_64-linux-musl/8.3.0/collect2
   1176 /usr/bin/make
    713 /usr/lib/gcc/x86_64-linux-gnu/9/cc1
    666 /usr/bin/as
    527 /usr/bin/gcc
    512 /root/heads/crossgcc/bin/x86_64-linux-musl-ar
    467 /usr/bin/ln
    449 /root/heads/build/gawk-4.2.1/gawk
    426 /usr/bin/basename
    329 /usr/bin/uname
    294 /root/heads/crossgcc/bin/i386-linux-musl-gcc
    293 /root/heads/crossgcc/bin/../libexec/gcc/i386-linux-musl/8.3.0/cc1
    289 /usr/bin/install
    286 /root/heads/crossgcc/bin/../lib/gcc/i386-linux-musl/8.3.0/../../../../i386-linux-musl/bin/as
    281 /usr/local/bin/make
    281 /usr/bin/wc
    279 /usr/sbin/make
    274 /usr/bin/echo
    268 /usr/bin/date
    266 /usr/local/sbin/make
    257 /usr/bin/tee
    253 /root/heads/crossgcc/bin/x86_64-linux-musl-ld
    212 /usr/bin/gnatgcc
    208 /usr/bin/ld
    192 /usr/lib/gcc/x86_64-linux-gnu/9/collect2
    187 /usr/bin/cmp
    180 /usr/bin/touch
    174 /usr/bin/awk
    171 /usr/bin/fmt
    171 /usr/bin/cp
    162 /usr/bin/sort
    137 /usr/bin/rmdir
    123 /usr/bin/chmod
    117 /root/heads/crossgcc/bin/x86_64-linux-musl-strip
    103 /usr/bin/hostname
    101 /usr/bin/stat
     93 /usr/bin/file
     89 /usr/bin/which
     82 /usr/bin/mktemp
     82 /usr/bin/ls
     77 /usr/bin/diff
     74 /usr/sbin/as
     71 /root/heads/crossgcc/bin/x86_64-linux-musl-nm
     65 /usr/bin/ar
     64 /usr/local/bin/as
     63 /usr/local/sbin/as
     62 /usr/bin/cut
     61 /usr/bin/g++
     60 /usr/bin/pkg-config
     45 /usr/bin/git
     44 /usr/bin/sha256sum
     43 /usr/bin/print
     39 /usr/bin/tar
     36 /usr/bin/ranlib
     35 /usr/bin/patch
     31 /usr/lib/gcc/x86_64-linux-gnu/9/cc1plus
     31 /usr/bin/wget
     30 /usr/bin/getconf
     30 /root/heads/install/bin/gpg-error-config
     30 ./configure
     29 /usr/bin/arch
     29 /bin/uname
     29 /bin/arch
     27 /usr/bin/m4
     24 /usr/bin/find
     23 /usr/bin/sleep
     23 /root/heads/crossgcc/bin/i386-linux-musl-ar
     22 /usr/bin/uniq
     21 /usr/lib/git-core/git
     19 /usr/bin/true
     18 /usr/bin/dd
     18 /root/heads/crossgcc/bin/x86_64-linux-musl-objdump
     18 /root/heads/build/slang-2.3.1a/autoconf/mkinsdir.sh
     18 /root/heads/build/linux-4.14.62/linux-qemu/tools/objtool//fixdep
     17 /usr/bin/bzip2
     16 /usr/bin/xargs
     15 /usr/sbin/mkdir
     15 /usr/local/bin/mkdir
     15 /usr/bin/objdump
     15 /usr/bin/mt
     15 /usr/bin/gawk
     15 ./conftest
     14 /usr/sbin/file
     13 /usr/sbin/which
     13 /usr/local/sbin/mkdir
     13 /usr/local/bin/rm
     13 /usr/local/bin/echo
     12 /usr/local/sbin/echo
     12 /usr/bin/x86_64-linux-gnu-gcc
     12 /usr/bin/gzip
     12 /bin/pwd
     11 qemu-coreboot/util/cbfstool/cbfstool
     11 /usr/sbin/rm
     11 /usr/bin/xz
     11 /usr/bin/sh
     11 /usr/bin/cc
     11 /root/heads/crossgcc/bin/i386-linux-musl-objcopy
     11 ./gpgrt-config
     11 ./gpg-error-config-old
     10 /usr/sbin/echo
     10 /usr/local/bin/which
      9 /usr/local/bin/gcc
      9 /usr/bin/bison
      9 /root/heads/install/bin/libassuan-config
      9 /root/heads/crossgcc/bin/x86_64-linux-musl-readelf
      9 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/echo
      8 /usr/local/sbin/file
      8 /usr/bin/install-info
      8 /usr/bin/fgrep
      7 /usr/sbin/gcc
      7 /usr/local/sbin/rm
      7 /usr/local/sbin/gnatgcc
      7 /usr/local/sbin/gcc
      7 /usr/local/bin/cat
      7 /usr/bin/od
      7 /bin/machine
      6 applets/usage
      6 /usr/sbin/gnatgcc
      6 /usr/sbin/cp
      6 /usr/sbin/cat
      6 /usr/local/sbin/which
      6 /usr/local/sbin/cat
      6 /usr/local/bin/file
      6 /usr/bin/nm
      6 /usr/bin/head
      6 /usr/bin/getopt
      6 /root/heads/crossgcc/bin/x86_64-linux-musl-objcopy
      6 /root/heads/crossgcc/bin/i386-linux-musl-ld.bfd
      6 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/gnatgcc
      6 /bin/ln
      6 ./util/getrevision.sh
      5 /usr/local/bin/gnatgcc
      5 /usr/local/bin/cp
      5 /usr/bin/flex
      5 /usr/bin/cpio
      5 /usr/bin/bash
      5 /root/heads/install/bin/npth-config
      5 /root/heads/install/bin/libgcrypt-config
      5 /root/heads/crossgcc/bin/i386-linux-musl-nm
      5 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/as
      5 ./bin/cpio-clean
      4 applets/applet_tables
      4 /usr/local/sbin/cp
      4 /usr/local/bin/touch
      4 /usr/convex/getsysinfo
      4 /usr/bin/sha1sum
      4 /usr/bin/oslevel
      4 /usr/bin/ldd
      4 /usr/bin/envsubst
      4 /root/heads/install/bin/ksba-config
      3 scripts/kconfig/conf
      3 qemu-coreboot/util/kconfig/conf
      3 /usr/sbin/true
      3 /usr/sbin/mv
      3 /usr/sbin/expr
      3 /usr/local/sbin/sh
      3 /usr/local/sbin/bzip2
      3 /usr/local/bin/sh
      3 /usr/local/bin/mv
      3 /usr/local/bin/expr
      3 /usr/bin/x86_64-linux-gnu-objdump
      3 /usr/bin/x86_64-linux-gnu-as
      3 /usr/bin/pwd
      3 /usr/bin/hostinfo
      3 /usr/bin/autom4te
      3 /root/heads/build/busybox-1.28.0/scripts/generate_BUFSIZ.sh
      3 /root/heads/build/busybox-1.28.0/scripts/gen_build_files.sh
      3 /bin/universe
      2 scripts/mod/modpost
      2 scripts/kallsyms
      2 qemu-coreboot/util/nvramtool/nvramtool
      2 autoconf/scripts/getsyslibs.sh
      2 arch/x86/entry/vdso/vdso2c
      2 applets/usage_compressed
      2 /usr/sbin/uname
      2 /usr/sbin/sh
      2 /usr/sbin/ln
      2 /usr/sbin/gzip
      2 /usr/sbin/bzip2
      2 /usr/local/sbin/true
      2 /usr/local/sbin/touch
      2 /usr/local/sbin/gzip
      2 /usr/local/sbin/getconf
      2 /usr/local/sbin/expr
      2 /usr/local/sbin/env
      2 /usr/local/bin/uname
      2 /usr/local/bin/true
      2 /usr/local/bin/sha256sum
      2 /usr/local/bin/gzip
      2 /usr/local/bin/bzip2
      2 /usr/lib/git-core/git-submodule
      2 /usr/bin/strip
      2 /usr/bin/gettext
      2 /usr/bin/env
      2 /usr/bin/autoconf
      2 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/iasl
      2 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/gzip
      2 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/clang
      2 /root/heads/build/busybox-1.28.0/scripts/trylink
      2 /bin/false
      2 ./scripts/asn1_compiler
      2 ./mkheader
      2 ./inittests
      2 ./a.out
      1 util/xcompile/xcompile
      1 util/genbuild_h/genbuild_h.sh
      1 scripts/mod/mk_elfconfig
      1 scripts/conmakehash
      1 scripts/basic/split-include
      1 qemu-coreboot/util/sconfig/sconfig
      1 qemu-coreboot/util/romcc/romcc
      1 qemu-coreboot/util/cbfstool/fmaptool
      1 lib/gen_crc32table
      1 bin/bin-to-hex
      1 autoconf/mkinsdir.sh
      1 arch/x86/tools/relocs
      1 arch/x86/boot/tools/build
      1 arch/x86/boot/mkcpustr
      1 arch/x86/boot/compressed/mkpiggy
      1 applets/usage_pod
      1 /usr/sbin/touch
      1 /usr/sbin/rmdir
      1 /usr/sbin/ldconfig
      1 /usr/sbin/install
      1 /usr/sbin/grep
      1 /usr/sbin/env
      1 /usr/sbin/chmod
      1 /usr/sbin/bash
      1 /usr/local/sbin/strip
      1 /usr/local/sbin/sha256sum
      1 /usr/local/sbin/ranlib
      1 /usr/local/sbin/mv
      1 /usr/local/sbin/ln
      1 /usr/local/sbin/install
      1 /usr/local/sbin/find
      1 /usr/local/bin/xz
      1 /usr/local/bin/flex
      1 /usr/lib/git-core/git-remote-https
      1 /usr/bin/pod2text
      1 /usr/bin/pod2man
      1 /usr/bin/pod2html
      1 /usr/bin/egrep
      1 /usr/bin/bc
      1 /usr/bin/automake
      1 /usr/bin/autoheader
      1 /sbin/ldconfig.real
      1 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/uname
      1 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin/powerpc64-linux-gnu-clang
      1 /root/heads/build/coreboot-4.8.1/util/crossgcc/xgcc/bin//iasl
      1 /root/heads/build/coreboot-4.8.1/qemu-coreboot/cbfstool
      1 /root/heads/build/busybox-1.28.0/scripts/mkconfigs
      1 /root/heads/bin/cpio-clean
      1 ./yat2m
      1 ./util/git-hooks/install.sh
      1 ./scripts/sortextable
      1 ./scripts/basic/bin2c
      1 ./runtest
      1 ./mkerrcodes
      1 ./gpg-error-config-test.sh
      1 ./gost-s-box
      1 ./autogen.sh
      1 ./asn1-gentables

[osresearch]

(some of those are spurious -- <unfinished ...> system calls, for example, are messing up my simplistic grep of the logs to find the successful execve() calls)

[Thrilleratplay]

@osresearch I am assuming commands executed within scripts would appear separately. Couldn't shell scripts be removed from this list to simplify it?

Additionally, I am trying to compile Heads and coreboot in Guix. One of the features is the ability to generate deep dependency graphs. I'll post that here as soon as I am able to.

[osresearch]

That deep dependency graph is quite neat. I was thinking about making a text version of a dependency tree, although my immediate goal was a little simpler: ensure that we knew which external tools we were using to help identify where reproducibility issues might be creeping in.

For instance, where are the 128 invocations of /usr/bin/as coming from? Or the 37 /usr/bin/g++ and /usr/bin/ranlib? On Xen I ran into an issue where older versions of ranlib would mess up elf headers for some of the special features.

We also have an issue with #892 - /usr/bin/pkg-config is invoked many times and causes host configuration issues to bleed into the build.

And I'm especially amused by the 10 executions of /usr/bin/mt; it is probably configure script, but who has even run a magnetic tape control program in the last decade?

[tlaurion]

@Thrilleratplay any update, blog post, pointers out to fix zlib problem outlined here and on slack guix-buildstack channel?

[Thrilleratplay]

@tlaurion Unfortunately, not really. I started down the path of extracting more and more Debian packages to be able to build crossgcc and it got to the point where it was obvious this defeated the purpose of using Guix. Ada released a new version a few weeks ago that may help overcome the bootstrapping issue. I haven't done much with it in the past month; I have been catching up with other obligations in different projects/life. Sorry. I'll try to get back to this soon.

[Thrilleratplay]

@tlaurion I had sometime last night to take a look at where I was with this. Basically, I keep going around in the same circle trying to resolve dependencies with Ada, Gnat, GCC and Zlib. A few projects have asked or seeked out Ada/Gnat for Guix (example 1 and the Guix IRC logs containing the word gnat). Based on what samplet says here, my problem may be that the AdaCore binaries are not static. This coupled with the fact that the last 32bit AdaCore release was in 2014 and using GCC 4.7.4, this may not be possible in a reasonable amount of time before the rapture/robot apocalypse. samplet also mentions creating a Ada/Gnat bootstrap using 26 Debian packages and a lot of patchelf. It may have been destined for this repo but it is empty. This was the other path I was heading down but seemed counterproductive to being able to create reproducible environments. This may be the only realistic option until a modern open source Ada crosscompiler is developed. I go back to using Debian binaries.