[Buildroot] [PATCH 09/34] reproducibility/libglib2: allow removing codegen

[Arnout Vandecappelle]

On 05/10/16 21:42, Gilles Chanteperdrix wrote:
> On Tue, May 10, 2016 at 01:40:30AM +0200, Arnout Vandecappelle wrote:
>> On 05/08/16 22:25, Gilles Chanteperdrix wrote:
>>> On Sat, May 07, 2016 at 11:04:25PM +0200, Arnout Vandecappelle wrote:
>>>> On 04/30/16 09:49, Gilles Chanteperdrix wrote:
>>>>> But this is not sufficient, compiling the python bytecode with the
>>>>> same interpreter in different environments yields different binaries,
>>>>
>>>>   Er, this is worrisome... You are saying that we don't have a chance in hell of
>>>> generating reproducible python bytecode?
>>>
>>> I am not a python specialist, but it seems to me four bytes in the
>>> python generated bytecode are the build timestamp, so unless there
>>> is a way to override it with SOURCE_DATE_EPOCH, I do not see that
>>> possible.
>>
>>   I've checked the docs. What is saved is the timestamp of the .py file, not the
>> build time.
>
> Mmmm I don't remember. I would run a compilation manually, twice at
> a different time, to make sure that the problem is only the file
> date.

  I did that - admittedly with just a few seconds difference. Both in 
python2.7.11 and python3.5, they were identical when compiling a second time, 
and different after touch'ing.

>
>>
>>   I think it would make sense to run 'touch -d @$(SOURCE_DATE_EPOCH)' on all
>> files after patching to capture this aspect. This was also proposed for
>> Fedora[1] (though there it was only for the .py files). Not sure what happened
>> with that proposal in the end.
>
> I think I tried running "touch" before compiling, unfortunately
> playing with file dates before running "make" is a bit like playing
> with fire. For instance with autotools based projects for which
> autoreconf is not run, the project may use versions of the autotools
> not installed on the user machine, and because of file dates may
> want to rerun autoconf or autmake and whine because the right
> versions are not available. Doing this only for .py files is much
> more reasonable.

  I tested this as well, with make-3.81 and make-4.04. Both of them do _not_ 
rebuild if the timestamps are identical. And since the idea is to use touch -d 
@$(SOURCE_DATE_EPOCH), all timestamps will be identical.

  But it does indeed mean that if a package has a generated file with an earlier 
date than the source files, it will now suddenly no longer be rebuilt.

  I don't know how the various other build systems (waf, scons) will behave. 
There is indeed a risk, but I'd say we deal with that when it happens. Actually 
I think we would prefer to detect such issues.

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF