NAME

    Test2::Harness - AKA 'yath' Yet Another Test Harness, A modern
    alternative to prove.

DESCRIPTION

    This is the primary documentation for the yath command, Test2::Harness,
    App::Yath and all other related components.

    The yath command is an alternative to the prove command and
    Test::Harness.

PLATFORM SUPPORT

    Test2::Harness/App::Yath is is focused on unix-like platforms. Most
    development happens on linux, but bsd, macos, etc should work fine as
    well.

    Patches are welcome for any/all platforms, but the primary author (Chad
    'Exodist' Granum) does not directly develop against non-unix platforms.

 WINDOWS

    Currently windows is not supported, and it is known that the package
    will not install on windows. Patches are be welcome, and it would be
    great if someone wanted to take on the windows-support role, but it is
    not a primary goal for the project.

QUICK START

    You can use yath to run all the tests for your repo:

        $ yath test

    Some important notes for those used to the prove command:

    yath is recursive by default.

      You do not need to add the -r flag to reach tests in subdirectories
      of t/ and t2/.

    The 'lib/', 'blib/lib', snd 'blib/arch' paths are added to your tests
    @INC for you.

      No need to add '-Ilib', '-l', or '-b', as these are added
      automatically. They can be disabled if desired, but in most cases you
      want them.

    Yath will run tests concurrently by default

      By default yath will run tests with multiple processes. If you have
      System::Info installed it will use half of your processors/cores,
      otherwise it defaults to 2 processes.

      Yoy can disable concurrency with the -j1 flag, or specify a custom
      concurrency value with -j#.

COMMAND LINE HELP

    There are a couple useful things to be aware of:

    yath help - Get a list of commands

      This will provide you with a list of available yath commands, and a
      brief description of what they do.

    yath COMMAND --help

    yath help COMMAND

      These are both effectively the same thing, they let you get command
      specific help.

    yath COMMAND --help=SECTION

      Sometimes a command may have an overwhelming number of options. You
      can filter it to specific sections to make it easier to find what you
      are looking for. At the very end of the help dialogue is a list of
      sections to help you get start.

    yath COMMAND [OPTIONS] --show-opts

    yath COMMAND [OPTIONS] --show-opts=GROUP

      This will show you what yath interpreted all your config and command
      line args to mean in a tree view. This is useful if you suspect a
      command line flag is not being handled properly.

CONFIGURATION FILES

    Yath will read from the following configuration files when you run it.

    Note: These should be located in your projects root directory, yath
    will search parent directories to find them.

    .yath.rc

      This should contain project specific flags you always want used
      regardless of the machine. This file SHOULD be commited to your
      project repository.

    .yath.user.rc

      This should contain user specific flags that you onyl want to apply
      when you run tests. This file SHOULD NOT be commited to yor project
      repository.

    The format of the config files is:

        # GLOBAL OPTIONS FOR ALL COMMANDS
        -D/path/to/my/spcial/libs
    
        [test]              # Options for the 'test' command
        -j32                # Always use 32 processes for concurrency
        -Irel(foo/bar)      # Always include this path relative to the location of the .rc file
        -Irel(foo/bar/*)    # Wildcard expanded to multiple -I.. options
    
        [start]
        ...

    Any option that is valid at the command line can be put into the .rc
    file.

    Anything listed before a command section applies to all commands.

    Anything after a [COMMAND] section applies only to that command

    rel(...) can be used to provide paths relative to the location of the
    .rc file.

    rel(..*) wildcards will be expanded

    # starts a comment

PRELOADING AND CONCURRENCY FOR FASTER TEST RUNS

    You can preload modules that are expensive to load, then yath will
    launch tests from these preloaded states. In some cases this can
    provide a massive speedup:

        yath test -PMoose -PList::Util -PScalar::Util

    In addition yath can run multiple concurrent jobs (specified with the
    -j# command line option.

        yath test -j16

    You can combine these for a compounding performance boost:

        yath test -j16 -PMoose

 BENCHMARKING WITH THE MOOSE TEST SUITE:

    No concurrency, no preload (85s):

          $ yath test -j1
          [...]
               File Count: 478
          Assertion Count: 19546
                Wall Time: 85.10 seconds
                 CPU Time: 53.33 seconds (usr: 3.61s | sys: 0.17s | cusr: 44.85s | csys: 4.70s)
                CPU Usage: 62%

    No concurrency, but preload (26s):

          $ yath test -j1 -PMoose
          [...]
               File Count: 478
          Assertion Count: 19545
                Wall Time: 26.84 seconds
                 CPU Time: 18.39 seconds (usr: 2.71s | sys: 0.58s | cusr: 12.45s | csys: 2.65s)
                CPU Usage: 68%

    Just concurrency, no preload (27s):

          $ yath test -j16
          [...]
               File Count: 478
          Assertion Count: 19546
                Wall Time: 27.25 seconds
                 CPU Time: 58.62 seconds (usr: 4.24s | sys: 0.18s | cusr: 49.73s | csys: 4.47s)
                CPU Usage: 215%

    Concurrency + Preload (7s):

          $ yath test -j16 -PMoose
          [...]
               File Count: 478
          Assertion Count: 19545
                Wall Time: 7.12 seconds
                 CPU Time: 18.26 seconds (usr: 2.14s | sys: 0.10s | cusr: 13.93s | csys: 2.09s)
                CPU Usage: 256%

    As you can see concurrency and preloading make a huge difference in
    test run times!

    See "ADVANCED CONCURRENCY" and "ADVANCED PRELOADING" for more
    information.

USING A WEB INTERFACE

    Note: It is better to create a standalone yath web server, rather than
    creating a new instance for each run, Documentation on doing that will
    be linked here when it is written.

        $ yath test --server

    This will launch a web server (usually on http://127.0.0.1:8080, but
    the url will be printed for you at the start and end of the run) that
    allows you to view the results and filter/inspect the output in full
    detail.

    NOTE: this requires installaction of one of the following sets of
    optional modules:

    DBD::Pg and DateTime::Format::Pg

      PostgreSQL, the database engine Test2::Harness is primarily written
      against.

      You can specify this one directly if you want to avoid database
      roulette:

          $ yath test --server=PostgreSQL

    DBD::SQLite and DateTime::Format::SQLite

      SQLite, the easiest option if you do not want to install a larger
      database engine.

      You can specify this one directly if you want to avoid database
      roulette:

          $ yath test --server=SQLite

    DBD::MariaDB, DBD::mysql and DateTime::Format::MySQL

      These modules will let you use direct MariaDB support instead of
      falling back to a generic MySQL implementation.

          $ yath test --server=MariaDB

    DBD::mysql and DateTime::Format::MySQL

      With these modules installed you can choose to use generic MySQL, or
      a specific flavor of MySQL such as Percona.

      You can specify this one directly if you want to avoid database
      roulette:

          $ yath test --server=MySQL
          $ yath test --server=Percona

      Note: Percona has no built-in UUID type, as such it will be slow to
      handle operations that require UUIDs such as looking up specific
      events.

 HARNESS DIRECTIVES INSIDE TESTS

    yath will recognise a number of directive comments placed near the top
    of test files. These directives should be placed after the #! line but
    before any real code.

    Real code is defined as any line that does not start with use, require,
    BEGIN, package, or #

    good example 1

          #!/usr/bin/perl
          # HARNESS-NO-FORK
      
          ...

    good example 2

          #!/usr/bin/perl
          use strict;
          use warnings;
      
          # HARNESS-NO-FORK
      
          ...

    bad example 1

          #!/usr/bin/perl
      
          # blah
      
          # HARNESS-NO-FORK
      
          ...

    bad example 2

          #!/usr/bin/perl
      
          print "hi\n";
      
          # HARNESS-NO-FORK
      
          ...

  HARNESS-NO-PRELOAD

        #!/usr/bin/perl
        # HARNESS-NO-PRELOAD

    Use this if your test will fail when modules are preloaded. This will
    tell yath to start a new perl process to run the script instead of
    forking with preloaded modules.

    Currently this implies HARNESS-NO-FORK, but that may not always be the
    case.

  HARNESS-NO-FORK

        #!/usr/bin/perl
        # HARNESS-NO-FORK

    Use this if your test file cannot run in a forked process, but instead
    must be run directly with a new perl process.

    This implies HARNESS-NO-PRELOAD.

  HARNESS-NO-STREAM

    yath usually uses the Test2::Formatter::Stream formatter instead of
    TAP. Some tests depend on using a TAP formatter. This option will make
    yath use Test2::Formatter::TAP or Test::Builder::Formatter.

  HARNESS-NO-IO-EVENTS

    yath can be configured to use the Test2::Plugin::IOEvents plugin. This
    plugin replaces STDERR and STDOUT in your test with tied handles that
    fire off proper Test2::Event's when they are printed to. Most of the
    time this is not an issue, but any fancy tests or modules which do
    anything with STDERR or STDOUT other than print may have really messy
    errors.

    Note: This plugin is disabled by default, so you only need this
    directive if you enable it globally but need to turn it back off for
    select tests.

  HARNESS-NO-TIMEOUT

    yath will usually kill a test if no events occur within a timeout
    (default 60 seconds). You can add this directive to tests that are
    expected to trip the timeout, but should be allowed to continue.

    NOTE: you usually are doing the wrong thing if you need to set this.
    See: HARNESS-TIMEOUT-EVENT.

  HARNESS-TIMEOUT-EVENT 60

    yath can be told to alter the default event timeout from 60 seconds to
    another value. This is the recommended alternative to
    HARNESS-NO-TIMEOUT

  HARNESS-TIMEOUT-POSTEXIT 15

    yath can be told to alter the default POSTEXIT timeout from 15 seconds
    to another value.

    Sometimes a test will fork producing output in the child while the
    parent is allowed to exit. In these cases we cannot rely on the
    original process exit to tell us when a test is complete. In cases
    where we have an exit, and partial output (assertions with no final
    plan, or a plan that has not been completed) we wait for a timeout
    period to see if any additional events come into

  HARNESS-DURATION-LONG

    This lets you tell yath that the test file is long-running. This is
    primarily used when concurrency is turned on in order to run longer
    tests earlier, and concurrently with shorter ones. There is also a yath
    option to skip all long tests.

    This duration is set automatically if HARNESS-NO-TIMEOUT is set.

  HARNESS-DURATION-MEDIUM

    This lets you tell yath that the test is medium.

    This is the default duration.

  HARNESS-DURATION-SHORT

    This lets you tell yath That the test is short.

  HARNESS-CATEGORY-ISOLATION

    This lets you tell yath that the test cannot be run concurrently with
    other tests. Yath will hold off and run these tests one at a time after
    all other tests.

  HARNESS-CATEGORY-IMMISCIBLE

    This lets you tell yath that the test cannot be run concurrently with
    other tests of this class. This is helpful when you have multiple tests
    which would otherwise have to be run sequentially at the end of the
    run.

    Yath prioritizes running these tests above HARNESS-CATEGORY-LONG.

  HARNESS-CATEGORY-GENERAL

    This is the default category.

  HARNESS-CONFLICTS-XXX

    This lets you tell yath that no other test of type XXX can be run at
    the same time as this one. You are able to set multiple conflict types
    and yath will honor them.

    XXX can be replaced with any type of your choosing.

    NOTE: This directive does not alter the category of your test. You are
    free to mark the test with LONG or MEDIUM in addition to this marker.

  HARNESS-JOB-SLOTS 2

  HARNESS-JOB-SLOTS 1 10

    Specify a range of job slots needed for the test to run. If set to a
    single value then the test will only run if it can have the specified
    number of slots. If given a range the test will require at least the
    lower number of slots, and use up to the maximum number of slots.

    Example with multiple lines.

          #!/usr/bin/perl
          # DASH and space are split the same way.
          # HARNESS-CONFLICTS-DAEMON
          # HARNESS-CONFLICTS  MYSQL
      
          ...

    Or on a single line.

          #!/usr/bin/perl
          # HARNESS-CONFLICTS DAEMON MYSQL
      
          ...

  HARNESS-RETRY-n

    This lets you specify a number (minimum n=1) of retries on test failure
    for a specific test. HARNESS-RETRY-1 means a failing test will be run
    twice and is equivalent to HARNESS-RETRY.

  HARNESS-NO-RETRY

    Use this to avoid this test being retried regardless of your retry
    settings.

ADVANCED CONCURRENCY

    You can design your tests to use concurrency internally that is managed
    by yath!

    You can sub-divide concurrency slots by specifying -j#:#. This will set
    the $ENV{T2_HARNESS_MY_JOB_CONCURRENCY} env var to the number of
    concurrency slots assigned to the test.

    Note: Tests are normally only assigned 1 concurrency slot unless they
    have the # HARNESS-JOB-SLOTS MIN or #HARNESS-JOB-SLOTS MIN MAX headers
    at the top of the file (they can be after the shbang or use statements,
    but must be befor eany other code).

    Here is an example of a test written to use anywhere from 1 to 5 slots
    depending on how much yath gives it. Implementation of wait() and
    start_child_process() is left to the reader.

        #!/usr/bin/perl
        use strict;
        use warnings;
        use Test2::V0;
        use Test2::IPC;
        # HARNESS-JOB-SLOTS 1 5
    
        my @kids;
    
        for my (1 .. $ENV{T2_HARNESS_MY_JOB_CONCURRENCY}) {
            push @kids => start_child_process(...);
        }
    
        wait($_) for @kids;
    
        done_testing;

ADVANCED PRELOADING

    You can create custom preload classes (still loaded with -PClass) that
    define advanced preload behavior:

        package MyPreload;
        use strict;
        use warnings;
    
        # Using this class will turn this into an advanced preload class
        use Test2::Harness::Preload;
    
        stage ONLY_MOOSE => sub {
            preload 'Moose';
        };
    
        stage ORGANIZATION_COMMON_MODULES => sub {
            eager();
            default();
    
            preload 'Moose';
            preload 'My::Common::Foo';
            preload 'My::Common::Bar';
    
            preload sub { ... };
    
            stage APP_A => sub {
                preload 'My::App::A';
    
                post_fork sub { do_this_before_each_test_after_fork() };
            };
    
            stage APP_B => sub {
                preload 'My::APP::B;
            };
        };
    
        1;

    Custom preload classes use the Test2::Harness::Preload module to set up
    some key meta-information, then you can define preload "stages".

    A "stage" is a process with a preloaded state waiting for tests to run,
    when it gets a test it will fork and the test will run in the fork.
    These stages/forks are smart and use goto::file to insure no stack
    frames or undesired state contamination will be introduced into your
    test process.

    Stages may be nested, that is you can build a parent stage, then have
    child stages forked from that stage that inherit the state but also
    make independent changes.

    There are also several hooks available to inject behavior pre-fork,
    post-fork and pre-launch.

 FUNCTIONS AVAILABLE TO PRELOADS

    stage NAME => sub { ... }

    stage("NAME", sub { ... })

      Defines a stage, call hooks or other methods within the sub.

    preload "Module::Name"

    preload sub { ... }

      This designates modules that should be loaded in any given stage.
      They will be loaded in order. You may also provide a sub to do the
      loading if a simple module name is not sufficient such as if you need
      localized env vars or other clever tricks.

    eager()

      This designates a stage as eager. If a stage is eager then it will
      run tests that usually ask for a nested/child stage. This is useful
      if a nested stage takes a long time to load and you want tests that
      ask for it to start anyway without waiting longer than they have to.

    default()

      This desginates a stage as the default one. This means it will be
      used if the test does not request a specific stage. There can only be
      1 default stage.

    pre_fork sub { ... }

      This sub will be called just before forking every time a test is
      forked off of the stage. This is run in the parent process.

    post_fork sub { ... }

      This will be called immedietly after forking every time a test if
      forked off of the stage. This is run in the child process.

    pre_launch sub { ... }

      This will be called in the child process after state has been
      manipulated just before execution is handed off to the test file.

 SPECIFYING WHICH PRELOADS SHOULD BE USED FOR WHICH TESTS

    The easiest way is to add a specific header comment to your test files:

        #!/usr/bin/perl
        use strict;
        use warnings
        # HARNESS-STAGE-MYSTAGE

    You can also configure plugins (See "Plugins") to assign tests to
    stages.

PLUGINS

    TODO: WRITE ME!

RENDERERS

    TODO: WRITE ME!

RESOURCES

    TODO: WRITE ME!

SOURCE

    The source code repository for Test2-Harness can be found at
    http://github.com/Test-More/Test2-Harness/.

MAINTAINERS

    Chad Granum <exodist@cpan.org>

AUTHORS

    Chad Granum <exodist@cpan.org>

COPYRIGHT

    Copyright Chad Granum <exodist7@gmail.com>.

    This program is free software; you can redistribute it and/or modify it
    under the same terms as Perl itself.

    See http://dev.perl.org/licenses/

