Update tests, and add documentation! (#7506)

This PR does the following:

* fix tests -- now, they're always built in the CMake scripts, and they work on VS2015
  *add a new flag, BUILD_TESTING, which allows one to turn off testing builds
* Add documentation for running tests

docs/index.md

@@ -21,6 +21,11 @@ Vcpkg helps you manage C and C++ libraries on Windows, Linux and MacOS. This too
 - [Portfile functions](maintainers/portfile-functions.md)
 - [Maintainer Guidelines](maintainers/maintainer-guide.md)
 
+### Tool Maintainer Help
+
+- [Testing](tool-maintainers/testing.md)
+- [Maintainer Guidelines](maintainers/maintainer-guide.md)
+
 ### Specifications
 
 - [Export](specifications/export-command.md)

docs/tool-maintainers/testing.md

@@ -0,0 +1,162 @@
+Testing
+=======
+
+Testing vcpkg is important whenever one makes changes to the tool itself, and
+writing new tests and keeping them up to date is also very important. If one's
+code is subtly broken, we'd rather find it out right away than a few weeks down
+the line when someone complains!
+
+Running Tests
+-------------
+
+Before anything else, we should know whether you can actually run the tests!
+All you should need is a way to build vcpkg -- anything will do! All you have to
+do is follow the guide 😄
+
+With `$VCPKG_DIRECTORY` being the directory where you have cloned vcpkg, create
+a build directory in `$VCPKG_DIRECTORY/toolsrc` (commonly named `out`), and
+`cd` into it. Make sure to clean it out if it already exists!
+
+```sh
+$ cmake .. -DCMAKE_BUILD_TYPE=Debug -G Ninja
+$ cmake --build .
+$ ./vcpkg-test # ./vcpkg-test [$SPECIFIC_TEST] for a specific set of tests
+$ # i.e., ./vcpkg-test [arguments]
+```
+
+If you make any modifications to `vcpkg`, you'll have to do the
+`cmake --build .` step again.
+
+Writing Tests
+-------------
+
+In your journey to write new tests, and to modify existing tests, reading the
+[Catch2 documentation] will be very helpful! Come back after reading those 😀
+
+You'll want to place your tests in one of the existing files, or, if it doesn't
+belong in any of those, in a [new file](#adding-new-test-files).
+
+The layout of these tests is as follows:
+
+```cpp
+// ... includes
+
+TEST_CASE("Name of test", "[filename without the .cpp]") {
+  // setup and the like
+  REQUIRE(some boolean expression);
+}
+
+// etc.
+```
+
+You want to give these test cases good, descriptive, unique names, like
+`SourceParagraph construct minimum` -- it doesn't need to be extremely clear
+english, and shorthand is good, but make sure it's clear what the test is from
+the name. For the latter parameter, known as "tags", you should at least put the
+name of the file which the test case is in -- e.g., in `arguments.cpp`, you'd
+tag all of the test cases with `[arguments]`.
+
+If you wish to add helper functions, make sure to place them in an anonymous
+namespace -- this will ensure that they don't trample over anybody else's
+space. Additionally, there are a few helper functions that live in
+`<vcpkg-test/util.h>` and `src/vcpkg-test/util.cpp` -- make sure to look into
+them so that you're not rewriting functionality.
+
+That should be all you need to know to start writing your own tests!
+Remember to check out the [Catch2 documentation]
+if you'd like to get more advanced with your tests,
+and good luck on your testing journey!
+
+Adding New Test Files
+---------------------
+
+Adding new test files should be easy and straightforward. All it requires is
+creating a new source file in `toolsrc/src/vcpkg-test`, and then rerunning
+`CMake` in order to pick up the glob changes.
+
+### Example
+
+Let's try writing a new test file called `example` (very creative, I know).
+
+First, we should create a file, `example.cpp`, in `toolsrc/src/vcpkg-test`:
+
+```cpp
+// vcpkg-test/example.cpp
+#include <vcpkg-test/catch.h>
+```
+
+This is the minimum file needed for tests; let's rebuild our CMake directory.
+You'll have to clean out the existing `out` directory for CMake to rerun
+globbing.
+
+```sh
+$ cmake .. -DCMAKE_BUILD_TYPE=Debug -G Ninja
+# ...
+-- Build files have been written to: $VCPKG_DIRECTORY/toolsrc/out
+$ cmake --build .
+[80/80] Linking CXX executable vcpkg.exe
+```
+
+Okay, now let's make sure this worked; add a test case to `example.cpp`:
+
+```cpp
+TEST_CASE("Example 1 - fail", "[example]") {
+  REQUIRE(false);
+}
+```
+
+Now build the tests again, and run them:
+
+```sh
+$ cmake --build .
+[2/2] Linking CXX executable vcpkg-test.exe
+$ ./vcpkg-test
+
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+vcpkg-test.exe is a Catch v2.9.1 host application.
+Run with -? for options
+
+-------------------------------------------------------------------------------
+Example 1 - fail
+-------------------------------------------------------------------------------
+$VCPKG_DIRECTORY/toolsrc/src/vcpkg-test/example.cpp(3)
+...............................................................................
+
+$VCPKG_DIRECTORY/toolsrc/src/vcpkg-test/example.cpp(14): FAILED:
+  REQUIRE( false )
+
+===============================================================================
+test cases:  102 |  101 passed | 1 failed
+assertions: 3611 | 3610 passed | 1 failed
+```
+
+Hopefully, that worked! It should compile correctly, and have one failing test.
+Now let's try a more complex test, after deleting the old one;
+
+```cpp
+// add #include <vcpkg/base/strings.h> to the top of the file
+namespace Strings = vcpkg::Strings;
+
+TEST_CASE("Example 2 - success", "[example]") {
+  std::string hello = "Hello";
+  REQUIRE(Strings::case_insensitive_ascii_equals(hello, "hELLo"));
+  REQUIRE_FALSE(Strings::case_insensitive_ascii_starts_with(hello, "E"));
+}
+```
+
+Now compile and build the tests, and this time let's only run our example tests:
+
+```sh
+$ cmake --build .
+[2/2] Linking CXX executable vcpkg-test.exe
+$ ./vcpkg-test [example]
+Filters: [example]
+===============================================================================
+All tests passed (2 assertions in 1 test case)
+```
+
+Hopefully you have one test running and succeeding! If you have that, you have
+succeeded at adding a new file to vcpkg's tests. Congratulations! Have fun on
+the rest of your journey 🐱‍👤😁
+
+[Catch2 documentation]: https://github.com/catchorg/Catch2/blob/master/docs/tutorial.md#top

scripts/bootstrap.sh

@@ -249,7 +249,7 @@ buildDir="$vcpkgRootDir/toolsrc/build.rel"
 rm -rf "$buildDir"
 mkdir -p "$buildDir"
 
-(cd "$buildDir" && CXX=$CXX "$cmakeExe" .. -DCMAKE_BUILD_TYPE=Release -G "Ninja" "-DCMAKE_MAKE_PROGRAM=$ninjaExe" "-DDEFINE_DISABLE_METRICS=$vcpkgDisableMetrics" "-DVCPKG_ALLOW_APPLE_CLANG=$vcpkgAllowAppleClang") || exit 1
+(cd "$buildDir" && CXX=$CXX "$cmakeExe" .. -DCMAKE_BUILD_TYPE=Release -G "Ninja" "-DCMAKE_MAKE_PROGRAM=$ninjaExe" "-DBUILD_TESTING=OFF" "-DDEFINE_DISABLE_METRICS=$vcpkgDisableMetrics" "-DVCPKG_ALLOW_APPLE_CLANG=$vcpkgAllowAppleClang") || exit 1
 (cd "$buildDir" && "$cmakeExe" --build .) || exit 1
 
 rm -rf "$vcpkgRootDir/vcpkg"

toolsrc/CMakeLists.txt

@@ -2,11 +2,16 @@ cmake_minimum_required(VERSION 3.3)
 
 project(vcpkg C CXX)
 
-enable_testing()
-
+OPTION(BUILD_TESTING "Option for enabling testing" ON)
 OPTION(DEFINE_DISABLE_METRICS "Option for disabling metrics" OFF)
 OPTION(VCPKG_ALLOW_APPLE_CLANG "Option for allowing apple clang" OFF)
 
+if (DEFINE_DISABLE_METRICS)
+    set(DISABLE_METRICS_VALUE "1")
+else()
+    set(DISABLE_METRICS_VALUE "0")
+endif()
+
 if(CMAKE_COMPILER_IS_GNUXX OR CMAKE_CXX_COMPILER_ID MATCHES "GNU")
     set(GCC 1)
 elseif(CMAKE_CXX_COMPILER_ID MATCHES "AppleClang")
@@ -27,46 +32,12 @@ else()
     message(FATAL_ERROR "Unknown compiler: ${CMAKE_CXX_COMPILER_ID}")
 endif()
 
-if(GCC OR CLANG)
-    add_compile_options(-std=c++1z)
-    if(WERROR)
-        add_compile_options(-Wall -Wno-unknown-pragmas -Werror)
-    endif()
-endif()
-
-file(GLOB_RECURSE VCPKGLIB_SOURCES src/vcpkg/*.cpp)
-file(GLOB_RECURSE VCPKGTEST_SOURCES src/vcpkg-test/*.cpp)
-
-if (DEFINE_DISABLE_METRICS)
-    set(DISABLE_METRICS_VALUE "1")
-else()
-    set(DISABLE_METRICS_VALUE "0")
-endif()
-
-add_library(vcpkglib OBJECT ${VCPKGLIB_SOURCES})
-add_executable(vcpkg src/vcpkg.cpp $<TARGET_OBJECTS:vcpkglib>)
-add_executable(vcpkg-test
-    EXCLUDE_FROM_ALL
-    ${VCPKGTEST_SOURCES}
-    $<TARGET_OBJECTS:vcpkglib>)
-
-foreach(TARGET_NAME vcpkglib vcpkg vcpkg-test)
-    target_compile_definitions(${TARGET_NAME} PRIVATE -DDISABLE_METRICS=${DISABLE_METRICS_VALUE})
-    target_include_directories(${TARGET_NAME} PRIVATE include)
-endforeach()
-
 set(THREADS_PREFER_PTHREAD_FLAG ON)
 find_package(Threads REQUIRED)
-target_link_libraries(vcpkg PRIVATE Threads::Threads)
-target_link_libraries(vcpkg-test PRIVATE Threads::Threads)
 
-foreach(TEST_NAME
-    arguments chrono dependencies files
-    paragraph plan specifier statusparagraphs
-    strings supports update
-)
-    add_test(${TEST_NAME} vcpkg-test [${TEST_NAME}])
-endforeach()
+add_definitions(-DDISABLE_METRICS=${DISABLE_METRICS_VALUE})
+include_directories(include)
+link_libraries(Threads::Threads)
 
 if(CLANG)
     include(CheckCXXSourceCompiles)
@@ -80,11 +51,32 @@ if(CLANG)
 endif()
 
 if(GCC OR (CLANG AND USES_LIBSTDCXX))
-    target_link_libraries(vcpkg PRIVATE stdc++fs)
-    target_link_libraries(vcpkg-test PRIVATE stdc++fs)
+    link_libraries(stdc++fs)
 elseif(CLANG)
-    target_link_libraries(vcpkg PRIVATE c++fs)
-    target_link_libraries(vcpkg-test PRIVATE c++fs)
+    link_libraries(c++fs)
+endif()
+
+if(GCC OR CLANG)
+    add_compile_options(-std=c++1z)
+    if(WERROR)
+        add_compile_options(-Wall -Wno-unknown-pragmas -Werror)
+    endif()
+endif()
+
+file(GLOB_RECURSE VCPKGLIB_SOURCES src/vcpkg/*.cpp)
+
+add_library(vcpkglib OBJECT ${VCPKGLIB_SOURCES})
+add_executable(vcpkg src/vcpkg.cpp $<TARGET_OBJECTS:vcpkglib>)
+
+if (BUILD_TESTING)
+    file(GLOB_RECURSE VCPKGTEST_SOURCES src/vcpkg-test/*.cpp)
+
+    enable_testing()
+    add_executable(vcpkg-test
+        ${VCPKGTEST_SOURCES}
+        $<TARGET_OBJECTS:vcpkglib>)
+
+    add_test(NAME default COMMAND vcpkg-test [${TEST_NAME}])
 endif()
 
 if(MSVC)

toolsrc/src/vcpkg-test/util.cpp

@@ -76,12 +76,13 @@ namespace vcpkg::Test
 
     static bool system_allows_symlinks()
     {
-#if defined(_WIN32)
-        if (!__cpp_lib_filesystem)
-        {
-            return false;
-        }
-
+#if FILESYSTEM_SYMLINK == FILESYSTEM_SYMLINK_NONE
+        return false;
+#elif FILESYSTEM_SYMLINK == FILESYSTEM_SYMLINK_UNIX
+        return true;
+#elif !defined(_WIN32) // FILESYSTEM_SYMLINK == FILESYSTEM_SYMLINK_STD
+        return true;
+#else
         HKEY key;
         bool allow_symlinks = true;
 
@@ -97,8 +98,6 @@ namespace vcpkg::Test
         if (status == ERROR_SUCCESS) RegCloseKey(key);
 
         return allow_symlinks;
-#else
-        return true;
 #endif
     }
 
@@ -125,8 +124,8 @@ namespace vcpkg::Test
     const bool SYMLINKS_ALLOWED = system_allows_symlinks();
     const fs::path TEMPORARY_DIRECTORY = internal_temporary_directory();
 
-#if FILESYSTEM_SYMLINK == FILSYSTEM_SYMLINK_NONE
-    constexpr inline char no_filesystem_message[] =
+#if FILESYSTEM_SYMLINK == FILESYSTEM_SYMLINK_NONE
+    constexpr char no_filesystem_message[] =
         "<filesystem> doesn't exist; on windows, we don't attempt to use the win32 calls to create symlinks";
 #endif