[Buildroot] [PATCH 3/3] package/dmalloc: don't use SSP

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998 at free.fr> writes:

 > dmalloc directly calls into $(LD) to generate a shared library our of
 > the static one.

 > To detet what command it should run, ./configure tries various
 > incantations of ld with various command line options until one does not
 > fail. One of those is (basically):
 >     $(LD) --whole-archive -o contest.o.t contest.a

 > This makes ./configure conclude what the command to link a shared
 > library in the Makefile should be, and thus stores that in a variable:
 >     shlinkargs='$(LD) --whole-archive -o $@'

 > ... which is then AC_SUBST()ed into Makefile.in with a rule like:

 >     $(SHLIB): $(LIBRARY)
 >         @shlinkargs@ $(LIRARY)

 > which once substiuted, gives:

 >     $(SHLIB): $(LIBRARY)
 >         $(LD) --whole-archive -o $@ $(LIRARY)

 > However, when SSP is enabled, the __stack_chk_fail_local and co symbols
 > are provided by additional libraries or object files, and that is the
 > responsibility of gcc to pass those when linking. But as dmalloc
 > directly calls ld, it misses those.

 > Changing dmalloc to use $(CC) is not trivial, however.

 > First, we can't pass LD=$(TARGET_CC), otherwise the whole package
 > explodes [0]: indeed --whole-archive is unknown to gcc, so it must be
 > passed as -Wl,--whole archive instead. So we'd need to add a new test
 > that uses $(CC), like so:
 >     $(CC) -Wl,--whole-archive -o contest.o.t contest.a

 > However, in that case, gcc does pass additional libs/objs (like, for
 > eample, the SSP ones) to the linker. But then those are also inc;luded
 > in the whole-archive section. This causes the linker to add all the
 > symbols form those libs/objs, even those not needed for SSP; on some
 > archs, like PPC, that may require floating point symbols (__muldiv3 et
 > al.) which are in another library, and thus the linked can't find them.

 > The proper solution wouild be to add -Wl,--no-whole-archive, but that
 > would have to be added _after_ the library we want to link, i.e.e we
 > should be able to evntually run:

 >     $(CC) -Wl,--whole-archive -o $@ $(LIRARY) -Wl,--no-whole-archive

 > That would require that we introduce a new variable that is added
 > _afgter_ the $(LIBRARY), e.g. @shlinkargs_post@ or so...

 > This is a bigger endeavour than we want to pursue...

 > Since dmalloc is a debugging utility, it is not supposed to go into
 > production builds, and during debugging, it would not be surprising that
 > it needs to poke around arrays to debug them.

 > So, we go the easier route: disable SSP altogether.

 > [0] with lots of nce colors, but that's not the point, is it?

 > Signed-off-by: Yann E. MORIN <yann.morin.1998 at free.fr>
 > Cc: Fabrice Fontaine <fontaine.fabrice at gmail.com>

Committed to 2022.11.x and 2022.02.x, thanks.

-- 
Bye, Peter Korsgaard