[Buildroot] [PATCH 1/1] support/dependencies: bump minimal tar version to 1.35

[Vincent Fazio]

All

On Wed, Nov 8, 2023 at 3:01 PM Arnout Vandecappelle <arnout at mind.be> wrote:

>   Hi Vincent,
>
> On 18/10/2023 16:11, Vincent Fazio wrote:
> > GNU tar 1.35 includes a breaking change [0] that changes the tar header
> > created for each regular file even for "stable" formats like pax (which
> > we use in support/download/helpers::mk_tar_gz).
> >
> > Previously, normal files had the devmajor/devminor fields of the header
> > filled with ASCII zero (0x30). These fields are now null which also
> > affects the checksum value in the header as well.
> >
> >    $ diff <(head -n 100 xxd.1.34.output) <(head -n 100 xxd.1.35.output)
> >    10c10
> >    < 00000090: 3037 3500 3031 3531 3637 0020 3000 0000  075.015167. 0...
> >    ---
> >    > 00000090: 3037 3500 3031 3337 3237 0020 3000 0000  075.013727. 0...
> >    21,22c21,22
> >    < 00000140: 0000 0000 0000 0000 0030 3030 3030 3030  .........0000000
> >    < 00000150: 0030 3030 3030 3030 0000 0000 0000 0000  .0000000........
> >    ---
> >    > 00000140: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> >    > 00000150: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> >
> > This has the consequence of causing hash mismatches for any tarball
> > created with a version prior to 1.35.
> >
> > Instead of reverting back to a host specific version (see 37a909cacf)
> > and adding a cap to the maximal tar version (see ec50e407be), just
> > establish that 1.35+ should be used going forward.
> >
> > This change requires an updated BR_FMT_VERSION for both git and svn.
> >
> > [0]
> https://git.savannah.gnu.org/cgit/tar.git/commit/?id=738de9ecdec45ebfeb999628742373b5f8253bd0
> >
> > Signed-off-by: Vincent Fazio <vfazio at gmail.com>
>
>   We discussed with 3 more maintainers (Peter, Romain and myself) and came
> to
> the conclusion that it is really too annoying that almost everybody would
> have
> to build host-tar (unconditionally!). At the moment, of the main distros,
> only
>


The host-tar package takes between 20-45 seconds to build. I don't see that
as overly burdensome on a build that can potentially take upwards of half
an hour.



> Fedora 39 (just released) and Arch Linux already have 1.35.
>
>

We're already in a situation where people are using the latest Arch and
encountering this (a colleague of Thomas in fact).
It's what drove the investigation in October and this patch. They were
using an older buildroot, so there's not much that can be done there.
It's not ideal, but we can document the limitation with distributions using
tar 1.35 and recommend utilizing a build container that uses a previous
version of tar.



>   The idea is to apply this "bump minimal tar" only when more major
> distros have
> switched to 1.35 or later. Or, perhaps with the additional time we find an
> alternative that is hopefully more stable. I was thinking of switching to
> sharutils, for example - last release is from 2015 so very stable! :-)
>
>

Based on testing Yann did back in the day
(cbe95b1a455bbefcaa90a08cf3dd1a590630921e) the --format=posix format has
been stable since at least GNU tar 1.27 released 10 years ago.
I did open an issue in GNU tar for this change and asked about the
guarantee of stability: https://savannah.gnu.org/bugs/index.php?64875
Pending the outcome of that issue, we can maybe not do anything for now
except recommend against using distros with tar 1.35

I personally think it's better to do the bump and account for the behavior
change now. The build of host-tar 1.35 will eventually not be required as
the new version is adopted by distros vs sticking with 1.34 and then slowly
forcing more builds of older tar on newer distros.



>
  So, for the time being, we can do the following:
>
> - in check-host-tar.sh, check that system tar < 1.35;
> - in package/tar, add a patch that reverts 738de9ecde (probably need to
> revert
> c3663838 as well).
>


I don't agree with reverting these commits in 1.35 as it causes
functionally different behavior between BR built tar and distribution built
tar on the same version.
The only way to guarantee previous/stable behavior would be to _always_ use
BR built tar with reverts, which you didn't want to do above.

It would also make the recipe a bit funky, where we're specifically
applying a patch to 1.35 for the host because we rely on the behavior or
earlier versions
but probably don't want to do that for the target.



>
>
>   Regarding ec50e407be: we're indeed in a similar situation as back then.
> It's
> not ideal to have a maximal version, but for now it's better than putting
> the
> minimum at 1.35.
>
>

We could set the host tar version to 1.34, but it sounded like there were
concerns about security? The only CVE I see at a glance is CVE-2022-48303.
Upstream
https://git.savannah.gnu.org/cgit/tar.git/commit/?id=3da78400eafcccb97e2f2fd4b227ea40d794ede8
Oddly, Debian hasn't ported the patch but Ubuntu has.


I obviously defer to the decision of the maintainers, but wanted to voice
my thoughts.


>
>   Regards,
>   Arnout
>
>
> > ---
> > This patch should be coordinated with other patches [1] that update the
> > BR_FMT_VERSION revision for git or svn as all of the tarball hashes will
> > need to be regenerated tree-wide.
> >
> > [1] https://patchwork.ozlabs.org/project/buildroot/list/?series=373742
> > ---
> >   package/pkg-download.mk                |  4 ++--
> >   support/dependencies/check-host-tar.sh | 10 ++++++----
> >   2 files changed, 8 insertions(+), 6 deletions(-)
> >
> > diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> > index e5cd83d859..d33eb2d811 100644
> > --- a/package/pkg-download.mk
> > +++ b/package/pkg-download.mk
> > @@ -20,8 +20,8 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
> >
> >   # Version of the format of the archives we generate in the
> corresponding
> >   # download backend:
> > -BR_FMT_VERSION_git = -br1
> > -BR_FMT_VERSION_svn = -br3
> > +BR_FMT_VERSION_git = -br2
> > +BR_FMT_VERSION_svn = -br4
> >
> >   DL_WRAPPER = support/download/dl-wrapper
> >
> > diff --git a/support/dependencies/check-host-tar.sh
> b/support/dependencies/check-host-tar.sh
> > index b7d607a47a..d56d0242a9 100755
> > --- a/support/dependencies/check-host-tar.sh
> > +++ b/support/dependencies/check-host-tar.sh
> > @@ -27,11 +27,13 @@ if [ -n "${version_bsd}" ] ; then
> >       exit 1
> >   fi
> >
> > -# Minimal version = 1.27 (previous versions do not correctly unpack
> archives
> > -# containing hard-links if the --strip-components option is used or
> create
> > -# different gnu long link headers for path elements > 100 characters).
> > +# Minimal version = 1.35
> > +# GNU tar upstream commit 738de9ecd introduced a "breaking" change that
> > +# affects tarballs regardless of mode (pax/ustar/v7) such that tarballs
> > +# generated with 1.35 will not hash to the same value as those generated
> > +# by previous versions of tar.
> >   major_min=1
> > -minor_min=27
> > +minor_min=35
> >
> >   if [ $major -lt $major_min ]; then
> >       # echo nothing: no suitable tar found
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.buildroot.org/pipermail/buildroot/attachments/20231109/4b1177d9/attachment-0001.html>