NAME

YAML::PP - YAML 1.2 processor

SYNOPSIS

WARNING: Most of the inner API is not stable yet.

Here are a few examples of the basic load and dump methods:

    use YAML::PP;
    my $ypp = YAML::PP->new;

    my $yaml = <<'EOM';
    --- # Document one is a mapping
    name: Tina
    age: 29
    favourite language: Perl

    --- # Document two is a sequence
    - plain string
    - 'in single quotes'
    - "in double quotes we have escapes! like \t and \n"
    - | # a literal block scalar
      line1
      line2
    - > # a folded block scalar
      this is all one
      single line because the
      linebreaks will be folded
    EOM

    my @documents = $ypp->load_string($yaml);
    my @documents = $ypp->load_file($filename);

    my $yaml = $ypp->dump_string($data1, $data2);
    $ypp->dump_file($filename, $data1, $data2);

    # The loader offers JSON::PP::Boolean, boolean.pm or
    # perl 1/'' (currently default) for booleans
    my $ypp = YAML::PP->new(boolean => 'JSON::PP');
    my $ypp = YAML::PP->new(boolean => 'boolean');
    my $ypp = YAML::PP->new(boolean => 'perl');

    # Enable perl data types and objects
    my $ypp = YAML::PP->new(schema => [qw/ + Perl /]);
    my $yaml = $yp->dump_string($data_with_perl_objects);

    # Legacy interface
    use YAML::PP qw/ Load Dump LoadFile DumpFile /;
    my @documents = Load($yaml);
    my @documents = LoadFile($filename);
    my @documents = LoadFile($filehandle);
    my $yaml = = Dump(@documents);
    DumpFile($filename, @documents);
    DumpFile($filenhandle @documents);

Some utility scripts, mostly useful for debugging:

    # Load YAML into a data structure and dump with Data::Dumper
    yamlpp-load < file.yaml

    # Load and Dump
    yamlpp-load-dump < file.yaml

    # Print the events from the parser in yaml-test-suite format
    yamlpp-events < file.yaml

    # Parse and emit events directly without loading
    yamlpp-parse-emit < file.yaml

    # Create ANSI colored YAML. Can also be useful for invalid YAML, showing
    # you the exact location of the error
    yamlpp-highlight < file.yaml

DESCRIPTION

YAML::PP is a modular YAML processor.

It aims to support YAML 1.2 and YAML 1.1. See http://yaml.org/. Some (rare) syntax elements are not yet supported and documented below.

YAML is a serialization language. The YAML input is called "YAML Stream". A stream consists of one or more "Documents", separated by a line with a document start marker ---. A document optionally ends with the document end marker ....

This allows one to process continuous streams additionally to a fixed input file or string.

The YAML::PP frontend will currently load all documents, and return only the first if called with scalar context.

The YAML backend is implemented in a modular way that allows one to add custom handling of YAML tags, perl objects and data types. The inner API is not yet stable. Suggestions welcome.

You can check out all current parse and load results from the yaml-test-suite here: https://perlpunk.github.io/YAML-PP-p5/test-suite.html

METHODS

new

    my $ypp = YAML::PP->new;
    # load booleans via boolean.pm
    my $ypp = YAML::PP->new( boolean => 'boolean' );
    # load booleans via JSON::PP::true/false
    my $ypp = YAML::PP->new( boolean => 'JSON::PP' );
    
    # use YAML 1.2 Failsafe Schema
    my $ypp = YAML::PP->new( schema => ['Failsafe'] );
    # use YAML 1.2 JSON Schema
    my $ypp = YAML::PP->new( schema => ['JSON'] );
    # use YAML 1.2 Core Schema
    my $ypp = YAML::PP->new( schema => ['Core'] );
    
    # Die when detecting cyclic references
    my $ypp = YAML::PP->new( cyclic_refs => 'fatal' );
    
    my $ypp = YAML::PP->new(
        boolean => 'JSON::PP',
        schema => ['Core'],
        cyclic_refs => 'fatal',
        indent => 4,
        header => 1,
        footer => 1,
        version_directive => 1,
    );

Options:

boolean

Values: perl (currently default), JSON::PP, boolean

schema

Default: ['Core']

Array reference. Here you can define what schema to use. Supported standard Schemas are: Failsafe, JSON, Core, YAML1_1.

To get an overview how the different Schemas behave, see https://perlpunk.github.io/YAML-PP-p5/schemas.html

Additionally you can add further schemas, for example Merge.

cyclic_refs

Default: 'allow' but will be switched to fatal in the future for safety!

Defines what to do when a cyclic reference is detected when loading.

    # fatal  - die
    # warn   - Just warn about them and replace with undef
    # ignore - replace with undef
    # allow  - Default
indent

Default: 2

Use that many spaces for indenting

Default: 1

Print document heaader ---

Default: 0

Print document footer ...

yaml_version

Since version 0.020

Default: 1.2

Note that in this case, a directive %YAML 1.1 will basically be ignored and everything loaded with the 1.2 Core Schema.

If you want to support both YAML 1.1 and 1.2, you have to specify that, and the schema (Core or YAML1_1) will be chosen automatically.

    my $yp = YAML::PP->new(
        yaml_version => ['1.2', '1.1'],
    );

This is the same as

    my $yp = YAML::PP->new(
        schema => ['+'],
        yaml_version => ['1.2', '1.1'],
    );

because the + stands for the default schema per version.

When loading, and there is no %YAML directive, 1.2 will be considered as default, and the Core schema will be used.

If there is a %YAML 1.1 directive, the YAML1_1 schema will be used.

Of course, you can also make 1.1 the default:

    my $yp = YAML::PP->new(
        yaml_version => ['1.1', '1.2'],
    );

You can also specify 1.1 only:

    my $yp = YAML::PP->new(
        yaml_version => ['1.1'],
    );

In this case also documents with %YAML 1.2 will be loaded with the YAML1_1 schema.

version_directive

Since version 0.020

Default: 0

Print Version Directive %YAML 1.2 (or %YAML 1.1) on top of each YAML document. It will use the first version specified in the yaml_version option.

preserve

Since version 0.021

Default: false

Preserving scalar styles is still experimental.

    use YAML::PP::Common qw/ PRESERVE_ORDER PRESERVE_SCALAR_STYLE /;

    # Preserve the order of hash keys
    my $yp = YAML::PP->new( preserve => PRESERVE_ORDER );

    # Preserve the quoting style of scalars
    my $yp = YAML::PP->new( preserve => PRESERVE_SCALAR_STYLE );

    # Preserve order and scalar style
    my $yp = YAML::PP->new( preserve => PRESERVE_ORDER | PRESERVE_SCALAR_STYLE );

Do NOT rely on the internal implementation of it.

If you load the following input:

    ---
    z: 1
    a: 2
    ---
    - plain
    - 'single'
    - "double"
    - |
      literal

    my $yp = YAML::PP->new( preserve => PRESERVE_ORDER | PRESERVE_SCALAR_STYLE );
    my ($hash, $styles) = $yp->load_file($file);

Then dumping it will return the same output. Only folded block scalars '>' cannot preserve the style yet.

When loading, hashes will be tied to an internal class (YAML::PP::Preserve::Hash) that keeps the key order.

Scalars will be returned as objects of an internal class (YAML::PP::Preserve::Scalar) with overloading. If you assign to such a scalar, the object will be replaced by a simple scalar.

    # assignment, style gets lost
    $styles->[1] .= ' append';

You can also pass 1 as a value. In this case all preserving options will be enabled, also if there are new options added in the future.

load_string

    my $doc = $ypp->load_string("foo: bar");
    my @docs = $ypp->load_string("foo: bar\n---\n- a");

Input should be Unicode characters.

So if you read from a file, you should decode it, for example with Encode::decode().

Note that in scalar context, load_string and load_file return the first document (like YAML::Syck), while YAML and YAML::XS return the last.

load_file

    my $doc = $ypp->load_file("file.yaml");
    my @docs = $ypp->load_file("file.yaml");

Strings will be loaded as unicode characters.

dump_string

    my $yaml = $ypp->dump_string($doc);
    my $yaml = $ypp->dump_string($doc1, $doc2);
    my $yaml = $ypp->dump_string(@docs);

Input strings should be Unicode characters.

Output will return Unicode characters.

So if you want to write that to a file (or pass to YAML::XS, for example), you typically encode it via Encode::encode().

dump_file

    $ypp->dump_file("file.yaml", $doc);
    $ypp->dump_file("file.yaml", $doc1, $doc2);
    $ypp->dump_file("file.yaml", @docs);

Input data should be Unicode characters.

dump

This will dump to a predefined writer. By default it will just use the YAML::PP::Writer and output a string.

    my $writer = MyWriter->new(\my $output);
    my $yp = YAML::PP->new(
        writer => $writer,
    );
    $yp->dump($data);

loader

Returns or sets the loader object, by default YAML::PP::Loader

dumper

Returns or sets the dumper object, by default YAML::PP::Dumper

schema

Returns or sets the schema object

default_schema

Creates and returns the default schema

FUNCTIONS

The functions Load, LoadFile, Dump and DumpFile are provided as a drop-in replacement for other existing YAML processors. No function is exported by default.

Note that in scalar context, Load and LoadFile return the first document (like YAML::Syck), while YAML and YAML::XS return the last.

Load
    use YAML::PP qw/ Load /;
    my $doc = Load($yaml);
    my @docs = Load($yaml);

Works like load_string.

LoadFile
    use YAML::PP qw/ LoadFile /;
    my $doc = LoadFile($file);
    my @docs = LoadFile($file);
    my @docs = LoadFile($filehandle);

Works like load_file.

Dump
    use YAML::PP qw/ Dump /;
    my $yaml = Dump($doc);
    my $yaml = Dump(@docs);

Works like dump_string.

DumpFile
    use YAML::PP qw/ DumpFile /;
    DumpFile($file, $doc);
    DumpFile($file, @docs);
    DumpFile($filehandle, @docs);

Works like dump_file.

PLUGINS

You can alter the behaviour of YAML::PP by using the following schema classes:

YAML::PP::Schema::Failsafe

One of the three YAML 1.2 official schemas

YAML::PP::Schema::JSON

One of the three YAML 1.2 official schemas.

YAML::PP::Schema::Core

One of the three YAML 1.2 official schemas. Default

YAML::PP::Schema::YAML1_1

Schema implementing the most common YAML 1.1 types

YAML::PP::Schema::Perl

Serializing Perl objects and types

YAML::PP::Schema::Binary

Serializing binary data

YAML::PP::Schema::Tie::IxHash

Deprecated. See option preserve

YAML::PP::Schema::Merge

YAML 1.1 merge keys for mappings

YAML::PP::Schema::Include

Include other YAML files via !include tags

To make the parsing process faster, you can plugin the libyaml parser with YAML::PP::LibYAML.

IMPLEMENTATION

The process of loading and dumping is split into the following steps:

    Load:

    YAML Stream        Tokens        Event List        Data Structure
              --------->    --------->        --------->
                lex           parse           construct


    Dump:

    Data Structure       Event List        YAML Stream
                --------->        --------->
                represent           emit

You can dump basic perl types like hashes, arrays, scalars (strings, numbers). For dumping blessed objects and things like coderefs have a look at YAML::PP::Perl/YAML::PP::Schema::Perl.

YAML::PP::Lexer

The Lexer is reading the YAML stream into tokens. This makes it possible to generate syntax highlighted YAML output.

Note that the API to retrieve the tokens will change.

YAML::PP::Parser

The Parser retrieves the tokens from the Lexer. The main YAML content is then parsed with the Grammar.

YAML::PP::Grammar
YAML::PP::Constructor

The Constructor creates a data structure from the Parser events.

YAML::PP::Loader

The Loader combines the constructor and parser.

YAML::PP::Dumper

The Dumper will delegate to the Representer

YAML::PP::Representer

The Representer will create Emitter events from the given data structure.

YAML::PP::Emitter

The Emitter creates a YAML stream.

YAML::PP::Parser

Still TODO:

Implicit collection keys
    ---
    [ a, b, c ]: value
Implicit mapping in flow style sequences
    ---
    [ a, b, c: d ]
    # equals
    [ a, b, { c: d } ]
Plain mapping keys ending with colons
    ---
    key ends with two colons::: value
Supported Characters

If you have valid YAML that's not parsed, or the other way round, please create an issue.

Line and Column Numbers

You will see line and column numbers in the error message. The column numbers might still be wrong in some cases.

Error Messages

The error messages need to be improved.

Unicode Surrogate Pairs

Currently loaded as single characters without validating

Possibly more

YAML::PP::Constructor

The Constructor now supports all three YAML 1.2 Schemas, Failsafe, JSON and Core. Additionally you can choose the schema for YAML 1.1 as YAML1_1.

Too see what strings are resolved as booleans, numbers, null etc. look at https://perlpunk.github.io/YAML-PP-p5/schema-examples.html.

You can choose the Schema like this:

    my $ypp = YAML::PP->new(schema => ['JSON']); # default is 'Core'

The Tags !!seq and !!map are still ignored for now.

It supports:

Handling of Anchors/Aliases

Like in modules like YAML, the Constructor will use references for mappings and sequences, but obviously not for scalars.

YAML::XS uses real aliases, which allows also aliasing scalars. I might add an option for that since aliasing is now available in pure perl.

Boolean Handling

You can choose between 'perl' (1/'', currently default), 'JSON::PP' and 'boolean'.pm for handling boolean types. That allows you to dump the data structure with one of the JSON modules without losing information about booleans.

Numbers

Numbers are created as real numbers instead of strings, so that they are dumped correctly by modules like JSON::PP or JSON::XS, for example.

Complex Keys

Mapping Keys in YAML can be more than just scalars. Of course, you can't load that into a native perl structure. The Constructor will stringify those keys with Data::Dumper instead of just returning something like HASH(0x55dc1b5d0178).

Example:

    use YAML::PP;
    use JSON::PP;
    my $ypp = YAML::PP->new;
    my $coder = JSON::PP->new->ascii->pretty->allow_nonref->canonical;
    my $yaml = <<'EOM';
    complex:
        ?
            ?
                a: 1
                c: 2
            : 23
        : 42
    EOM
    my $data = $yppl->load_string($yaml);
    say $coder->encode($data);
    __END__
    {
       "complex" : {
          "{'{a => 1,c => 2}' => 23}" : 42
       }
    }

TODO:

Parse Tree

I would like to generate a complete parse tree, that allows you to manipulate the data structure and also dump it, including all whitespaces and comments. The spec says that this is throwaway content, but I read that many people wish to be able to keep the comments.

YAML::PP::Dumper, YAML::PP::Emitter

The Dumper should be able to dump strings correctly, adding quotes whenever a plain scalar would look like a special string, like true, or when it contains or starts with characters that are not allowed.

Most strings will be dumped as plain scalars without quotes. If they contain special characters or have a special meaning, they will be dumped with single quotes. If they contain control characters, including <"\n">, they will be dumped with double quotes.

It will recognize JSON::PP::Boolean and boolean.pm objects and dump them correctly.

Numbers which also have a PV flag will be recognized as numbers and not as strings:

    my $int = 23;
    say "int: $int"; # $int will now also have a PV flag

That means that if you accidentally use a string in numeric context, it will also be recognized as a number:

    my $string = "23";
    my $something = $string + 0;
    print $yp->dump_string($string);
    # will be emitted as an integer without quotes!

The layout is like libyaml output:

    key:
    - a
    - b
    - c
    ---
    - key1: 1
      key2: 2
      key3: 3
    ---
    - - a1
      - a2
    - - b1
      - b2

WHY

All the available parsers and loaders for Perl are behaving differently, and more important, aren't conforming to the spec. YAML::XS is doing pretty well, but libyaml only handles YAML 1.1 and diverges a bit from the spec. The pure perl loaders lack support for a number of features.

I was going over YAML.pm issues end of 2016, integrating old patches from rt.cpan.org and creating some pull requests myself. I realized that it would be difficult to patch YAML.pm to parse YAML 1.1 or even 1.2, and it would also break existing usages relying on the current behaviour.

In 2016 Ingy döt Net initiated two really cool projects:

"YAML TEST SUITE"
"YAML EDITOR"

These projects are a big help for any developer. So I got the idea to write my own parser and started on New Year's Day 2017. Without the test suite and the editor I would have never started this.

I also started another YAML Test project which allows one to get a quick overview of which frameworks support which YAML features:

"YAML TEST MATRIX"

YAML TEST SUITE

https://github.com/yaml/yaml-test-suite

It contains about 230 test cases and expected parsing events and more. There will be more tests coming. This test suite allows you to write parsers without turning the examples from the Specification into tests yourself. Also the examples aren't completely covering all cases - the test suite aims to do that.

The suite contains .tml files, and in a separate 'data' release you will find the content in separate files, if you can't or don't want to use TestML.

Thanks also to Felix Krause, who is writing a YAML parser in Nim. He turned all the spec examples into test cases.

YAML EDITOR

This is a tool to play around with several YAML parsers and loaders in vim.

https://github.com/yaml/yaml-editor

The project contains the code to build the frameworks (16 as of this writing) and put it into one big Docker image.

It also contains the yaml-editor itself, which will start a vim in the docker container. It uses a lot of funky vimscript that makes playing with it easy and useful. You can choose which frameworks you want to test and see the output in a grid of vim windows.

Especially when writing a parser it is extremely helpful to have all the test cases and be able to play around with your own examples to see how they are handled.

YAML TEST MATRIX

I was curious to see how the different frameworks handle the test cases, so, using the test suite and the docker image, I wrote some code that runs the tests, manipulates the output to compare it with the expected output, and created a matrix view.

https://github.com/perlpunk/yaml-test-matrix

You can find the latest build at http://matrix.yaml.io

As of this writing, the test matrix only contains valid test cases. Invalid ones will be added.

CONTRIBUTORS

Ingy döt Net

Ingy is one of the creators of YAML. In 2016 he started the YAML Test Suite and the YAML Editor. He also made useful suggestions on the class hierarchy of YAML::PP.

Felix "flyx" Krause

Felix answered countless questions about the YAML Specification.

SEE ALSO

YAML
YAML::XS
YAML::Syck
YAML::Tiny
YAML::PP::LibYAML
YAML::LibYAML::API

SPONSORS

The Perl Foundation https://www.perlfoundation.org/ sponsored this project (and the YAML Test Suite) with a grant of 2500 USD in 2017-2018.

COPYRIGHT AND LICENSE

Copyright 2017-2020 by Tina Müller

This library is free software and may be distributed under the same terms as perl itself.