cross compile - adding standard link directories in a toolchain file

Hello. I try to cross-compile Linux(x86_64)->Linux(aarch64). And add link directory on toolchain file. Cmake version 3.22.0

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

set(CMAKE_SYSROOT /aarch64-image)

set(CMAKE_C_COMPILER /usr/bin/aarch64-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER /usr/bin/aarch64-linux-gnu-g++)

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

link_directories(/directory1)

I try generators: Ninja and Makefiles.

The problem is that cmake correcly add this /directory1 to CMAKE_CXX_IMPLICIT_LINK_DIRECTORIES in generated file build/CMakeFiles/3.22.0/CMakeCXXCompiler.cmake. But this link directory is absent in Ninja or Makefile, so link of application failed.

But if we change in toolchain file directory1->directory2 and just rerun cmake configure in build directory cmake .. - than magically directory2 appears in link commands.

Very like that is bug.

Is any way to add additional link directories in toolchain file?