Debian Bug report logs - #1096129
buildd.d.o: some build-depends from incoming.d.o don't end up on snapshot.d.o

Display info messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: submit@bugs.debian.org

Subject: buildd.d.o: some build-depends from incoming.d.o don't end up on snapshot.d.o

Date: Sun, 16 Feb 2025 16:09:33 +0000

[Message part 1 (text/plain, inline)]

Source: buildd.debian.org
Severity: important
User: reproducible-builds@lists.alioth.debian.org
Usertags: infrastructure
X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org

hi,

we observed that some builds use build-depends from incoming.d.o
which then are not made available on snapshot.d.o (nor ftp.d.o)
because another upload of them happens during the same dinstall.

#1096112 "several binNMUs for bugs found on reproduce.debian.net amd64.reproduce.debian.net"
shows this happens surprisingly often, in the bug we request
92 binNMUs on amd64 and 66 on i386 (and other archs are still 
to be investigated.)

I've no idea how to best fix that but I wanted to file a bug so
we have a place to discuss solutions.


-- 
cheers,
	Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

“We live in capitalism. Its power seems inescapable. So did the divine right
 of kings. Any human power can be resisted and changed by human beings.
 Resistance and change often begin in art, and very often in our art, the art
 of words.” ― Ursula K. Le Guin

[signature.asc (application/pgp-signature, inline)]

Message #10 received at 1096129@bugs.debian.org (full text, mbox, reply):

From: Philipp Kern <pkern@debian.org>

To: Holger Levsen <holger@layer-acht.org>, 1096129@bugs.debian.org, control@bugs.debian.org

Subject: Re: Bug#1096129: buildd.d.o: some build-depends from incoming.d.o don't end up on snapshot.d.o

Date: Sun, 16 Feb 2025 17:16:25 +0100

reassign 1096129 ftp.debian.org
thanks

On 2/16/25 5:09 PM, Holger Levsen wrote:
> Source: buildd.debian.org
> Severity: important
> User: reproducible-builds@lists.alioth.debian.org
> Usertags: infrastructure
> X-Debbugs-Cc: reproducible-bugs@lists.alioth.debian.org
> 
> hi,
> 
> we observed that some builds use build-depends from incoming.d.o
> which then are not made available on snapshot.d.o (nor ftp.d.o)
> because another upload of them happens during the same dinstall.
> 
> #1096112 "several binNMUs for bugs found on reproduce.debian.net amd64.reproduce.debian.net"
> shows this happens surprisingly often, in the bug we request
> 92 binNMUs on amd64 and 66 on i386 (and other archs are still 
> to be investigated.)
> 
> I've no idea how to best fix that but I wanted to file a bug so
> we have a place to discuss solutions.

I don't think that's anything the buildds can fix, if the archive is not
publishing all intermediate products.

The only way on the buildds would be build failures until the next
dinstall, which would be backwards IMO.

I think the root cause here is that snapshot is not integrated with the
archive and only sees intermediate states and no incoming. So maybe it's
a snapshot bug, but my feeling is that the ftp-team would need to
propose on how to fix that.

Kind regards
Philipp Kern

Message #17 received at 1096129@bugs.debian.org (full text, mbox, reply):

From: Jochen Sprickerhof <jspricke@debian.org>

To: Philipp Kern <pkern@debian.org>, 1096129@bugs.debian.org

Cc: Holger Levsen <holger@layer-acht.org>

Subject: Re: Bug#1096129: buildd.d.o: some build-depends from incoming.d.o don't end up on snapshot.d.o

Date: Wed, 19 Feb 2025 10:41:18 +0100

[Message part 1 (text/plain, inline)]

* Philipp Kern <pkern@debian.org> [2025-02-16 17:16]:
>> we observed that some builds use build-depends from incoming.d.o
>> which then are not made available on snapshot.d.o (nor ftp.d.o)
>> because another upload of them happens during the same dinstall.
>>
>> #1096112 "several binNMUs for bugs found on reproduce.debian.net amd64.reproduce.debian.net"
>> shows this happens surprisingly often, in the bug we request
>> 92 binNMUs on amd64 and 66 on i386 (and other archs are still
>> to be investigated.)
>>
>> I've no idea how to best fix that but I wanted to file a bug so
>> we have a place to discuss solutions.
>
>I don't think that's anything the buildds can fix, if the archive is not
>publishing all intermediate products.
>
>The only way on the buildds would be build failures until the next
>dinstall, which would be backwards IMO.
>
>I think the root cause here is that snapshot is not integrated with the
>archive and only sees intermediate states and no incoming. So maybe it's
>a snapshot bug, but my feeling is that the ftp-team would need to
>propose on how to fix that.

I agree that it would be nice to get intermediate products on 
snapshot.d.o and I think there are two ways to do it.

1. Support incoming.d.o in snapshot.d.o. I don't know much about the 
technical details but at least it sounds ambitious given the temporary 
nature of incoming.

2. Integrate all intermediate products from incoming into the archive. 
Given that incoming could hold multiple package version between a 
dinstall I think that would mean adding all of them to the archive and 
keep them for at least a dinstall. Again not sure about the technical 
details but maybe dinstall could clean up old packages first and then 
add new ones from incoming.

Note that this bug is this is quiet a problem, we currently have 190 
source packages in arch:all that can't be reproduced and we can't binNMU 
easily:

https://all.reproduce.debian.net/stats/#packages-missing-on-metasnap-(maybe-temporary)

So maybe the step backward would actually be a step in the right 
direction till we can solve this in a better way?

Cheers Jochen

[signature.asc (application/pgp-signature, inline)]

Message #22 received at 1096129@bugs.debian.org (full text, mbox, reply):

From: Philipp Kern <pkern@debian.org>

To: Jochen Sprickerhof <jspricke@debian.org>

Cc: 1096129@bugs.debian.org, Holger Levsen <holger@layer-acht.org>

Subject: Re: Bug#1096129: buildd.d.o: some build-depends from incoming.d.o don't end up on snapshot.d.o

Date: Wed, 19 Feb 2025 11:17:18 +0100

On 2025-02-19 10:41, Jochen Sprickerhof wrote:
> * Philipp Kern <pkern@debian.org> [2025-02-16 17:16]:
>>> we observed that some builds use build-depends from incoming.d.o
>>> which then are not made available on snapshot.d.o (nor ftp.d.o)
>>> because another upload of them happens during the same dinstall.
>>> 
>>> #1096112 "several binNMUs for bugs found on reproduce.debian.net 
>>> amd64.reproduce.debian.net"
>>> shows this happens surprisingly often, in the bug we request
>>> 92 binNMUs on amd64 and 66 on i386 (and other archs are still
>>> to be investigated.)
>>> 
>>> I've no idea how to best fix that but I wanted to file a bug so
>>> we have a place to discuss solutions.
>> 
>> I don't think that's anything the buildds can fix, if the archive is 
>> not
>> publishing all intermediate products.
>> 
>> The only way on the buildds would be build failures until the next
>> dinstall, which would be backwards IMO.
>> 
>> I think the root cause here is that snapshot is not integrated with 
>> the
>> archive and only sees intermediate states and no incoming. So maybe 
>> it's
>> a snapshot bug, but my feeling is that the ftp-team would need to
>> propose on how to fix that.
> 
> I agree that it would be nice to get intermediate products on 
> snapshot.d.o and I think there are two ways to do it.
> 
> 1. Support incoming.d.o in snapshot.d.o. I don't know much about the 
> technical details but at least it sounds ambitious given the temporary 
> nature of incoming.

wanna-build gets a trigger whenever incoming.d.o updates. snapshot.d.o 
could get the same access and slurp it up from there.

> 2. Integrate all intermediate products from incoming into the archive. 
> Given that incoming could hold multiple package version between a 
> dinstall I think that would mean adding all of them to the archive and 
> keep them for at least a dinstall. Again not sure about the technical 
> details but maybe dinstall could clean up old packages first and then 
> add new ones from incoming.

I'd strongly prefer that in addition, I think. But the nature of 
snapshot means that it'd need to be a full archive, so maybe that would 
only work for ingesting packages, not archive files.

> Note that this bug is this is quiet a problem, we currently have 190 
> source packages in arch:all that can't be reproduced and we can't 
> binNMU easily:
> 
> https://all.reproduce.debian.net/stats/#packages-missing-on-metasnap-(maybe-temporary)
> 
> So maybe the step backward would actually be a step in the right 
> direction till we can solve this in a better way?

I disagree with this. Reproducible builds, as important as they are, 
should not hold back Debian development velocity.

Kind regards
Philipp Kern

Message #27 received at 1096129@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: Philipp Kern <pkern@debian.org>

Cc: Jochen Sprickerhof <jspricke@debian.org>, 1096129@bugs.debian.org

Subject: Re: Bug#1096129: buildd.d.o: some build-depends from incoming.d.o don't end up on snapshot.d.o

Date: Thu, 20 Feb 2025 00:31:05 +0000

[Message part 1 (text/plain, inline)]

On Wed, Feb 19, 2025 at 11:17:18AM +0100, Philipp Kern wrote:
> > 2. Integrate all intermediate products from incoming into the archive.
> > Given that incoming could hold multiple package version between a
> > dinstall I think that would mean adding all of them to the archive and
> > keep them for at least a dinstall. Again not sure about the technical
> > details but maybe dinstall could clean up old packages first and then
> > add new ones from incoming.
> I'd strongly prefer that in addition, I think. But the nature of snapshot
> means that it'd need to be a full archive, so maybe that would only work for
> ingesting packages, not archive files.
 
that's a different implementation than what Jochen suggested.

> > Note that this bug is this is quiet a problem, we currently have 190
> > source packages in arch:all that can't be reproduced and we can't binNMU
> > easily:
> > https://all.reproduce.debian.net/stats/#packages-missing-on-metasnap-(maybe-temporary)
> > So maybe the step backward would actually be a step in the right
> > direction till we can solve this in a better way?
> I disagree with this. Reproducible builds, as important as they are, should
> not hold back Debian development velocity.

I agree that Reproducible builds should not hold back Debian development
velocity but I don't think they have to. We just need a way to publish
incoming, which I think we need to do anyway.


-- 
cheers,
	Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

“Bitcoin was supposed to demonstrate the power of a true free market. Instead
 it's full of scams, rent-seekers, theft, useless for real purchases and
 accelerates climate change. Mission accomplished.” Adam Chalmers (@adam_chal)

[signature.asc (application/pgp-signature, inline)]

Message #32 received at 1096129@bugs.debian.org (full text, mbox, reply):

From: Ivo De Decker <ivodd@debian.org>

To: Jochen Sprickerhof <jspricke@debian.org>

Cc: Philipp Kern <pkern@debian.org>, 1096129@bugs.debian.org, Holger Levsen <holger@layer-acht.org>

Subject: Re: Bug#1096129: buildd.d.o: some build-depends from incoming.d.o don't end up on snapshot.d.o

Date: Mon, 9 Jun 2025 21:36:58 +0000

Control: tags -1 patch

Hi,

On Wed, Feb 19, 2025 at 10:41:18AM +0100, Jochen Sprickerhof wrote:
> I agree that it would be nice to get intermediate products on snapshot.d.o
> and I think there are two ways to do it.

[...]

> 2. Integrate all intermediate products from incoming into the archive. Given
> that incoming could hold multiple package version between a dinstall I think
> that would mean adding all of them to the archive and keep them for at least
> a dinstall. Again not sure about the technical details but maybe dinstall
> could clean up old packages first and then add new ones from incoming.

I submitted a MR to keep all binaries for at least 1 day, which should allow
them to end up in the ftp-master archive and snapshot.

https://salsa.debian.org/ftp-team/dak/-/merge_requests/287

Cheers,

Ivo

Message #39 received at 1096129@bugs.debian.org (full text, mbox, reply):

From: Holger Levsen <holger@layer-acht.org>

To: Ivo De Decker <ivodd@debian.org>

Cc: Jochen Sprickerhof <jspricke@debian.org>, Philipp Kern <pkern@debian.org>, 1096129@bugs.debian.org

Subject: Re: Bug#1096129: buildd.d.o: some build-depends from incoming.d.o don't end up on snapshot.d.o

Date: Tue, 10 Jun 2025 15:12:55 +0000

[Message part 1 (text/plain, inline)]

On Mon, Jun 09, 2025 at 09:36:58PM +0000, Ivo De Decker wrote:
> I submitted a MR to keep all binaries for at least 1 day, which should allow
> them to end up in the ftp-master archive and snapshot.
> https://salsa.debian.org/ftp-team/dak/-/merge_requests/287

whooohooo, thanks a lot!

Now we only need to make sure this gets deployed early in the forky circle...!


-- 
cheers,
	Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

People complaining about Debian changing something, please queue to the left.
People complaining about Debian moving too slowly please queue to the right.
Thank you. (@ekuber)

[signature.asc (application/pgp-signature, inline)]

Message #46 received at 1096129@bugs.debian.org (full text, mbox, reply):

From: Jochen Sprickerhof <jspricke@debian.org>

To: Ivo De Decker <ivodd@debian.org>, 1096129@bugs.debian.org

Cc: Philipp Kern <pkern@debian.org>, Holger Levsen <holger@layer-acht.org>

Subject: Re: Bug#1096129: buildd.d.o: some build-depends from incoming.d.o don't end up on snapshot.d.o

Date: Wed, 29 Oct 2025 17:22:42 +0100

[Message part 1 (text/plain, inline)]

Hi Ivo,

thanks so much for your work on this issue!

Do you plan to continue the work or could we help?

Cheers Jochen

* Ivo De Decker <ivodd@debian.org> [2025-06-09 21:36]:
>Control: tags -1 patch
>
>Hi,
>
>On Wed, Feb 19, 2025 at 10:41:18AM +0100, Jochen Sprickerhof wrote:
>> I agree that it would be nice to get intermediate products on snapshot.d.o
>> and I think there are two ways to do it.
>
>[...]
>
>> 2. Integrate all intermediate products from incoming into the archive. Given
>> that incoming could hold multiple package version between a dinstall I think
>> that would mean adding all of them to the archive and keep them for at least
>> a dinstall. Again not sure about the technical details but maybe dinstall
>> could clean up old packages first and then add new ones from incoming.
>
>I submitted a MR to keep all binaries for at least 1 day, which should allow
>them to end up in the ftp-master archive and snapshot.
>
>https://salsa.debian.org/ftp-team/dak/-/merge_requests/287
>
>Cheers,
>
>Ivo

[signature.asc (application/pgp-signature, inline)]

Message #51 received at 1096129-done@bugs.debian.org (full text, mbox, reply):

From: Joerg Jaspert <noreply@salsa.debian.org>

To: 1096129-done@bugs.debian.org

Subject: Bug#1096129 fixed in ftp.debian.org

Date: Sun, 16 Nov 2025 22:00:32 +0000

Hello,

Bug #1096129 in ftp.debian.org reported by you has been fixed in the Git repository.
You can see the commit message below and you can check the diff of the fix at:

https://salsa.debian.org/ftp-team/dak/-/commit/49eefa7f05b3362542c05802ff5e80dd7836c39e

------------------------------------------------------------------------
dak/dominate.py: keep binaries for at least 1 day

Don't remove binaries that have been in the archive for less than 1 day.

When multiple version of the same binary are uploaded without a dinstall
between them, the older versions are never published to the ftp-master archive.
They are only published in the buildd archive. This means they aren't available
in shapshot.debian.org either. Some of these binaries show up in buildinfo
files of other uploads, making it impossible to reproduce the binaries from
those uploads.

This change keeps all binaries around for at least a day, to allow them to be
published in the ftp-master archive and show up on snapshot.debian.org.

This does not happen for suites with build queues:
- avoid changing the behavior of dak dominate for (old)stable
- binaries from these suites enter the archive via suites with build queues
  anyway

Closes: #1096129
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/1096129

Message #52 received at 1096129-done@bugs.debian.org (full text, mbox, reply):

From: Joerg Jaspert <noreply@salsa.debian.org>

To: 1096129-done@bugs.debian.org

Subject: Bug#1096129 fixed in ftp.debian.org

Date: Mon, 17 Nov 2025 21:33:35 +0000

Hello,

Bug #1096129 in ftp.debian.org reported by you has been fixed in the Git repository.
You can see the commit message below and you can check the diff of the fix at:

https://salsa.debian.org/ftp-team/dak/-/commit/49eefa7f05b3362542c05802ff5e80dd7836c39e

------------------------------------------------------------------------
dak/dominate.py: keep binaries for at least 1 day

Don't remove binaries that have been in the archive for less than 1 day.

When multiple version of the same binary are uploaded without a dinstall
between them, the older versions are never published to the ftp-master archive.
They are only published in the buildd archive. This means they aren't available
in shapshot.debian.org either. Some of these binaries show up in buildinfo
files of other uploads, making it impossible to reproduce the binaries from
those uploads.

This change keeps all binaries around for at least a day, to allow them to be
published in the ftp-master archive and show up on snapshot.debian.org.

This does not happen for suites with build queues:
- avoid changing the behavior of dak dominate for (old)stable
- binaries from these suites enter the archive via suites with build queues
  anyway

Closes: #1096129
------------------------------------------------------------------------

(this message was generated automatically)
-- 
Greetings

https://bugs.debian.org/1096129

Send a report that this bug log contains spam.