[vcpkg] Track hashes of cmake script helpers in consuming portfiles (#12254)

* [vcpkg] Track hashes of cmake script helpers in consuming portfiles

* [vcpkg] Format

* [vcpkg_find_acquire_program] Use cpkg_from_sourceforge for FLEX and BISON

Co-authored-by: Robert Schumacher <roschuma@microsoft.com>

docs/maintainers/portfile-functions.md

@@ -21,8 +21,8 @@
 - [vcpkg\_configure\_make](vcpkg_configure_make.md)
 - [vcpkg\_configure\_meson](vcpkg_configure_meson.md)
 - [vcpkg\_copy\_pdbs](vcpkg_copy_pdbs.md)
-- [vcpkg\_copy\_tool\_dependencies](vcpkg_copy_tool_dependencies.md)
 - [vcpkg\_copy\_tools](vcpkg_copy_tools.md)
+- [vcpkg\_copy\_tool\_dependencies](vcpkg_copy_tool_dependencies.md)
 - [vcpkg\_download\_distfile](vcpkg_download_distfile.md)
 - [vcpkg\_execute\_build\_process](vcpkg_execute_build_process.md)
 - [vcpkg\_execute\_required\_process](vcpkg_execute_required_process.md)
@@ -31,10 +31,12 @@
 - [vcpkg\_fail\_port\_install](vcpkg_fail_port_install.md)
 - [vcpkg\_find\_acquire\_program](vcpkg_find_acquire_program.md)
 - [vcpkg\_fixup\_cmake\_targets](vcpkg_fixup_cmake_targets.md)
+- [vcpkg\_fixup\_pkgconfig](vcpkg_fixup_pkgconfig.md)
 - [vcpkg\_from\_bitbucket](vcpkg_from_bitbucket.md)
 - [vcpkg\_from\_git](vcpkg_from_git.md)
 - [vcpkg\_from\_github](vcpkg_from_github.md)
 - [vcpkg\_from\_gitlab](vcpkg_from_gitlab.md)
+- [vcpkg\_from\_sourceforge](vcpkg_from_sourceforge.md)
 - [vcpkg\_install\_cmake](vcpkg_install_cmake.md)
 - [vcpkg\_install\_gn](vcpkg_install_gn.md)
 - [vcpkg\_install\_make](vcpkg_install_make.md)

docs/maintainers/vcpkg_build_make.md

@@ -4,16 +4,26 @@ Build a linux makefile project.
 
 ## Usage:
 ```cmake
-vcpkg_build_make([TARGET <target>])
+vcpkg_build_make([BUILD_TARGET <target>]
+                 [ADD_BIN_TO_PATH]
+                 [ENABLE_INSTALL])
+                 [LOGFILE_ROOT <logfileroot>])
 ```
 
-### TARGET
+### BUILD_TARGET
-The target passed to the configure/make build command (`./configure/make/make install`). If not specified, no target will
+The target passed to the make build command (`./make <target>`). If not specified, the 'all' target will
 be passed.
 
 ### ADD_BIN_TO_PATH
 Adds the appropriate Release and Debug `bin\` directories to the path during the build such that executables can run against the in-tree DLLs.
 
+### ENABLE_INSTALL
+IF the port supports the install target use vcpkg_install_make() instead of vcpkg_build_make()
+
+### BUILD_TARGET
+The target passed to the make build command (`./make <target>`). If not specified, the 'all' target will
+be passed.
+
 ## Notes:
 This command should be preceeded by a call to [`vcpkg_configure_make()`](vcpkg_configure_make.md).
 You can use the alias [`vcpkg_install_make()`](vcpkg_configure_make.md) function if your CMake script supports the

docs/maintainers/vcpkg_common_definitions.md

@@ -16,6 +16,7 @@ VCPKG_TARGET_IMPORT_LIBRARY_PREFIX       import library prefix for target (same
 VCPKG_TARGET_IMPORT_LIBRARY_SUFFIX       import library suffix for target (same as CMAKE_IMPORT_LIBRARY_SUFFIX)
 VCPKG_FIND_LIBRARY_PREFIXES              target dependent prefixes used for find_library calls in portfiles
 VCPKG_FIND_LIBRARY_SUFFIXES              target dependent suffixes used for find_library calls in portfiles
+VCPKG_SYSTEM_LIBRARIES                   list of libraries are provide by the toolchain and are not managed by vcpkg
 ```
 
 CMAKE_STATIC_LIBRARY_(PREFIX|SUFFIX), CMAKE_SHARED_LIBRARY_(PREFIX|SUFFIX) and CMAKE_IMPORT_LIBRARY_(PREFIX|SUFFIX) are defined for the target

docs/maintainers/vcpkg_configure_make.md

@@ -7,9 +7,6 @@ Configure configure for Debug and Release builds of a project.
 vcpkg_configure_make(
     SOURCE_PATH <${SOURCE_PATH}>
     [AUTOCONFIG]
-    [DISABLE_AUTO_HOST]
-    [DISABLE_AUTO_DST]
-    [GENERATOR]
     [NO_DEBUG]
     [SKIP_CONFIGURE]
     [PROJECT_SUBPATH <${PROJ_SUBPATH}>]
@@ -28,10 +25,6 @@ By convention, this is usually set in the portfile as the variable `SOURCE_PATH`
 ### PROJECT_SUBPATH
 Specifies the directory containing the ``configure`/`configure.ac`.
 By convention, this is usually set in the portfile as the variable `SOURCE_PATH`.
-Should use `GENERATOR NMake` first.
-
-### NO_DEBUG
-This port doesn't support debug mode.
 
 ### SKIP_CONFIGURE
 Skip configure process
@@ -39,21 +32,8 @@ Skip configure process
 ### AUTOCONFIG
 Need to use autoconfig to generate configure file.
 
-### DISABLE_AUTO_HOST
-Don't set host automatically, the default value is `i686`.
-If use this option, you will need to set host manually.
-
-### DISABLE_AUTO_DST
-Don't set installation path automatically, the default value is `${CURRENT_PACKAGES_DIR}` and `${CURRENT_PACKAGES_DIR}/debug`
-If use this option, you will need to set dst path manually.
-
-### GENERATOR
-Specifies the precise generator to use.
-NMake: nmake(windows) make(unix)
-MAKE: make(windows) make(unix)
-
 ### PRERUN_SHELL
-Script that needs to be called before configuration
+Script that needs to be called before configuration (do not use for batch files which simply call autoconf or configure)
 
 ### OPTIONS
 Additional options passed to configure during the configuration.

docs/maintainers/vcpkg_fixup_pkgconfig.md

@@ -0,0 +1,42 @@
+# vcpkg_fixup_pkgconfig
+
+Fix common paths in *.pc files and make everything relativ to $(prefix)
+
+## Usage
+```cmake
+vcpkg_fixup_pkgconfig(
+    [RELEASE_FILES <PATHS>...]
+    [DEBUG_FILES <PATHS>...]
+    [SYSTEM_LIBRARIES <NAMES>...]
+)
+```
+
+## Parameters
+### RELEASE_FILES
+Specifies a list of files to apply the fixes for release paths.
+Defaults to every *.pc file in the folder ${CURRENT_PACKAGES_DIR} without ${CURRENT_PACKAGES_DIR}/debug/
+
+### DEBUG_FILES
+Specifies a list of files to apply the fixes for debug paths.
+Defaults to every *.pc file in the folder ${CURRENT_PACKAGES_DIR}/debug/
+
+### SYSTEM_PACKAGES
+If the *.pc file contains system packages outside vcpkg these need to be listed here.
+Since vcpkg checks the existence of all required packages within vcpkg.
+
+### SYSTEM_LIBRARIES
+If the *.pc file contains system libraries outside vcpkg these need to be listed here.
+VCPKG checks every -l flag for the existence of the required library within vcpkg.
+
+### IGNORE_FLAGS
+If the *.pc file contains flags in the lib field which are not libraries. These can be listed here
+
+## Notes
+Still work in progress. If there are more cases which can be handled here feel free to add them
+
+## Examples
+
+Just call vcpkg_fixup_pkgconfig() after any install step which installs *.pc files.
+
+## Source
+[scripts/cmake/vcpkg_fixup_pkgconfig.cmake](https://github.com/Microsoft/vcpkg/blob/master/scripts/cmake/vcpkg_fixup_pkgconfig.cmake)

docs/maintainers/vcpkg_from_sourceforge.md

@@ -42,6 +42,11 @@ The REPO is `soxr`, REF is not exist, and the FILENAME is `soxr-0.1.3-Source.tar
 ### SHA512
 The SHA512 hash that should match the archive.
 
+### WORKING_DIRECTORY
+If specified, the archive will be extracted into the working directory instead of `${CURRENT_BUILDTREES_DIR}/src/`.
+
+Note that the archive will still be extracted into a subfolder underneath that directory (`${WORKING_DIRECTORY}/${REF}-${HASH}/`).
+
 ### PATCHES
 A list of patches to be applied to the extracted sources.
 

scripts/cmake/vcpkg_find_acquire_program.cmake

@@ -244,33 +244,28 @@ function(vcpkg_find_acquire_program VAR)
     set(URL "https://github.com/mesonbuild/meson/archive/0.54.2.zip")
     set(ARCHIVE "meson-0.54.2.zip")
     set(HASH 8d19110bad3e6a223d1d169e833b746b884ece9cd23d2539ec02dccb5cd0c56542414b7afc0f7f2adcec9d957e4120d31f41734397aa0a7ee7f9c29ebdc9eb4c)
-  elseif(VAR MATCHES "FLEX")
+  elseif(VAR MATCHES "FLEX" OR VAR MATCHES "BISON")
     if(CMAKE_HOST_WIN32)
+      set(SOURCEFORGE_ARGS
+        REPO winflexbison
+        FILENAME winflexbison-2.5.16.zip
+        SHA512 0a14154bff5d998feb23903c46961528f8ccb4464375d5384db8c4a7d230c0c599da9b68e7a32f3217a0a0735742242eaf3769cb4f03e00931af8640250e9123
+        NO_REMOVE_ONE_LEVEL
+        WORKING_DIRECTORY "${DOWNLOADS}/tools/winflexbison"
+      )
+      if(VAR MATCHES "FLEX")
         set(PROGNAME win_flex)
-      set(SUBDIR win_flex-2.5.16)
+      else()
-      set(PATHS ${DOWNLOADS}/tools/win_flex/${SUBDIR})
+        set(PROGNAME win_bison)
-      set(URL "https://sourceforge.net/projects/winflexbison/files/winflexbison-2.5.16.zip/download")
+      endif()
-      set(ARCHIVE "win_flex_bison-2.5.16.zip")
+      set(PATHS ${DOWNLOADS}/tools/winflexbison/0a14154bff-a8cf65db07)
-      set(HASH 0a14154bff5d998feb23903c46961528f8ccb4464375d5384db8c4a7d230c0c599da9b68e7a32f3217a0a0735742242eaf3769cb4f03e00931af8640250e9123)
       if(NOT EXISTS "${PATHS}/data/m4sugar/m4sugar.m4")
         file(REMOVE_RECURSE "${PATHS}")
       endif()
-    else()
+    elseif(VAR MATCHES "FLEX")
       set(PROGNAME flex)
       set(APT_PACKAGE_NAME flex)
       set(BREW_PACKAGE_NAME flex)
-    endif()
-  elseif(VAR MATCHES "BISON")
-    if(CMAKE_HOST_WIN32)
-      set(PROGNAME win_bison)
-      set(SUBDIR win_bison-2.5.16)
-      set(PATHS ${DOWNLOADS}/tools/win_bison/${SUBDIR})
-      set(URL "https://sourceforge.net/projects/winflexbison/files/winflexbison-2.5.16.zip/download")
-      set(ARCHIVE "win_flex_bison-2.5.16.zip")
-      set(HASH 0a14154bff5d998feb23903c46961528f8ccb4464375d5384db8c4a7d230c0c599da9b68e7a32f3217a0a0735742242eaf3769cb4f03e00931af8640250e9123)
-      if(NOT EXISTS "${PATHS}/data/m4sugar/m4sugar.m4")
-        file(REMOVE_RECURSE "${PATHS}")
-      endif()
     else()
       set(PROGNAME bison)
       set(APT_PACKAGE_NAME bison)
@@ -378,6 +373,12 @@ function(vcpkg_find_acquire_program VAR)
       message(FATAL_ERROR "Could not find ${PROGNAME}. Please install it via your package manager${EXAMPLE}")
     endif()
 
+    if(DEFINED SOURCEFORGE_ARGS)
+      # Locally change editable to suppress re-extraction each time
+      set(_VCPKG_EDITABLE 1)
+      vcpkg_from_sourceforge(OUT_SOURCE_PATH SFPATH ${SOURCEFORGE_ARGS})
+      unset(_VCPKG_EDITABLE)
+    else()
       vcpkg_download_distfile(ARCHIVE_PATH
           URLS ${URL}
           SHA512 ${HASH}
@@ -415,6 +416,7 @@ function(vcpkg_find_acquire_program VAR)
           )
         endif()
       endif()
+    endif()
 
     if(DEFINED POST_INSTALL_COMMAND)
       vcpkg_execute_required_process(

scripts/cmake/vcpkg_from_sourceforge.cmake

@@ -42,6 +42,11 @@
 ## ### SHA512
 ## The SHA512 hash that should match the archive.
 ##
+## ### WORKING_DIRECTORY
+## If specified, the archive will be extracted into the working directory instead of `${CURRENT_BUILDTREES_DIR}/src/`.
+##
+## Note that the archive will still be extracted into a subfolder underneath that directory (`${WORKING_DIRECTORY}/${REF}-${HASH}/`).
+##
 ## ### PATCHES
 ## A list of patches to be applied to the extracted sources.
 ##
@@ -61,7 +66,7 @@
 
 function(vcpkg_from_sourceforge)
     set(booleanValueArgs DISABLE_SSL NO_REMOVE_ONE_LEVEL)
-    set(oneValueArgs OUT_SOURCE_PATH REPO REF SHA512 FILENAME)
+    set(oneValueArgs OUT_SOURCE_PATH REPO REF SHA512 FILENAME WORKING_DIRECTORY)
     set(multipleValuesArgs PATCHES)
     cmake_parse_arguments(_vdus "${booleanValueArgs}" "${oneValueArgs}" "${multipleValuesArgs}" ${ARGN})
 
@@ -77,6 +82,12 @@ function(vcpkg_from_sourceforge)
         message(FATAL_ERROR "The sourceforge repository must be specified.")
     endif()
 
+    if(DEFINED _vdus_WORKING_DIRECTORY)
+        set(WORKING_DIRECTORY WORKING_DIRECTORY "${_vdus_WORKING_DIRECTORY}")
+    else()
+        set(WORKING_DIRECTORY)
+    endif()
+
     if (_vdus_DISABLE_SSL)
         set(URL_PROTOCOL http:)
     else()
@@ -181,6 +192,7 @@ function(vcpkg_from_sourceforge)
         ARCHIVE "${ARCHIVE}"
         REF "${SANITIZED_REF}"
         ${NO_REMOVE_ONE_LEVEL}
+        ${WORKING_DIRECTORY}
         PATCHES ${_vdus_PATCHES}
     )
 

toolsrc/include/vcpkg/vcpkgpaths.h

@@ -83,6 +83,7 @@ namespace vcpkg
         bool is_valid_triplet(Triplet t) const;
         const std::vector<std::string> get_available_triplets_names() const;
         const std::vector<TripletFile>& get_available_triplets() const;
+        const std::map<std::string, std::string>& get_cmake_script_hashes() const;
         const fs::path get_triplet_file_path(Triplet triplet) const;
 
         fs::path original_cwd;

toolsrc/src/vcpkg/build.cpp

@@ -836,12 +836,16 @@ namespace vcpkg::Build
         abi_tag_entries.emplace_back("powershell", paths.get_tool_version("powershell-core"));
 #endif
 
-        abi_tag_entries.emplace_back(
+        auto& helpers = paths.get_cmake_script_hashes();
-            "vcpkg_fixup_cmake_targets",
+        auto portfile_contents =
-            vcpkg::Hash::get_file_hash(VCPKG_LINE_INFO,
+            fs.read_contents(port_dir / fs::u8path("portfile.cmake")).value_or_exit(VCPKG_LINE_INFO);
-                                       fs,
+        for (auto&& helper : helpers)
-                                       paths.scripts / "cmake" / "vcpkg_fixup_cmake_targets.cmake",
+        {
-                                       Hash::Algorithm::Sha1));
+            if (Strings::case_insensitive_ascii_contains(portfile_contents, helper.first))
+            {
+                abi_tag_entries.emplace_back(helper.first, helper.second);
+            }
+        }
 
         abi_tag_entries.emplace_back("post_build_checks", "2");
         std::vector<std::string> sorted_feature_list = action.feature_list;

toolsrc/src/vcpkg/vcpkgpaths.cpp

@@ -2,6 +2,7 @@
 
 #include <vcpkg/base/expected.h>
 #include <vcpkg/base/files.h>
+#include <vcpkg/base/hash.h>
 #include <vcpkg/base/system.debug.h>
 #include <vcpkg/base/system.process.h>
 #include <vcpkg/base/util.h>
@@ -90,6 +91,7 @@ namespace vcpkg
 
             Lazy<std::vector<VcpkgPaths::TripletFile>> available_triplets;
             Lazy<std::vector<Toolset>> toolsets;
+            Lazy<std::map<std::string, std::string>> cmake_script_hashes;
 
             Files::Filesystem* fs_ptr;
 
@@ -286,6 +288,26 @@ If you wish to silence this error and use classic mode, you can:
         });
     }
 
+    const std::map<std::string, std::string>& VcpkgPaths::get_cmake_script_hashes() const
+    {
+        return m_pimpl->cmake_script_hashes.get_lazy([this]() -> std::map<std::string, std::string> {
+            auto& fs = this->get_filesystem();
+            std::map<std::string, std::string> helpers;
+            auto files = fs.get_files_non_recursive(this->scripts / fs::u8path("cmake"));
+            auto common_functions = fs::u8path("vcpkg_common_functions");
+            for (auto&& file : files)
+            {
+                auto stem = file.stem();
+                if (stem != common_functions)
+                {
+                    helpers.emplace(stem.u8string(),
+                                    Hash::get_file_hash(VCPKG_LINE_INFO, fs, file, Hash::Algorithm::Sha1));
+                }
+            }
+            return helpers;
+        });
+    }
+
     const fs::path VcpkgPaths::get_triplet_file_path(Triplet triplet) const
     {
         return m_pimpl->m_triplets_cache.get_lazy(