[Buildroot] [PATCH 2/2] package/pkg-utils: teach per-package-rsync to copy or hardlink dest

[Yann E. MORIN]

Hervé, All,

On 2023-10-18 17:38 +0200, Herve Codina via buildroot spake thusly:
> On Wed, 18 Oct 2023 17:20:47 +0200
> "Yann E. MORIN" <yann.morin.1998 at free.fr> wrote:
> > Hervé, All
> > > > --- a/package/pkg-utils.mk
> > > > +++ b/package/pkg-utils.mk
> > > > @@ -214,10 +214,19 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
> > > >  # $1: space-separated list of packages to rsync from
> > > >  # $2: 'host' or 'target'
> > > >  # $3: destination directory
> > > > +# $4: literal "copy" or "hardlink" to copy or hardlink files from src to dest
> > > >  define per-package-rsync
> > > >  	mkdir -p $(3)
> > > >  	$(foreach pkg,$(1),\
> > > > -		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
> > > > +		rsync -a \
> > > > +			--hard-links \  
> > > You preserve hard links (--hard-links) in all cases (copy and hardlink).
> > > In case of copy, is it correct to preserve hard links ?  
> > Yes, --hard-links is:
> > 
> >     This tells rsync to look for hard-linked files in the source and
> >     link together the corresponding files on the destination.
> In case of 'copy', you keep hard-links already present in source and so the
> destination (final host/target dirs) is not a full copy.

Ah, it does not hard-link something in dest into src. The hardlinks are
only in the destination. See below...

> Some hard-links can be present and post-build scripts can still corrupt some
> per-package sources.
> 
> I don't understand why keeping hard-links on this last copy is needed and I
> probably miss something...

That is because we can have hard-links in host or target. For example,
git will install a lot of hard-links in /usr/libexec/git-core/
(excerpt):

    $ ls -li per-package/git/target/usr/libexec/git-core/
    9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-var
    9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-verify-commit
    9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-verify-pack
    9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-verify-tag
    9588770 -rwxr-xr-x 141 ymorin ymorin 3318828 Oct 16 17:45 git-version

    $ ls -li target/usr/libexec/git-core/
    9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-var
    9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-verify-commit
    9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-verify-pack
    9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-verify-tag
    9600190 -rwxr-xr-x 141 ymorin ymorin 2832860 Oct 16 17:45 git-version

As you can see, in git's PPD, git tools are hard-links, and in target/
they also are hard-links. But the inode is different, so the hard-links
in target/ are different from the hard-links in the PPD.

However, in all the PPDs of packages that have git as a dependency, the
hard-links would all be to the same inode, 9588770.

(The size delta is because the files in target/ are stripped.)

So, in case of "copy", this is actually a copy, that keeps existing
hard-links in the source, as new hard-links in the destination. Without
--hard-links, the files in target/usr/libexec/git-core/ would all have a
different inode, as they would be different files, and that would
tremendously increase the filesystem size (squashfs would notice, and
would store the content only once, but would still create as many inodes
as needed, though).

Regards,
Yann E. MORIN.

> Regards,
> Hervé
> 
> > 
> > Regards,
> > Yann E. MORIN.
> > 
> > > > +			$(if $(filter hardlink,$(4)), \
> > > > +				--link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/, \
> > > > +				$(if $(filter copy,$(4)), \
> > > > +					$(empty), \
> > > > +					$(error per-package-rsync can only "copy" or "hardlink", not "$(4)") \
> > > > +				) \
> > > > +			) \
> > > >  			$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
> > > >  			$(3)$(sep))
> > > >  endef  
> > > 
> > > [...]
> > > 
> _______________________________________________
> buildroot mailing list
> buildroot at buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'