|
libsgfc++ 2.0.1
A C++ library that uses SGFC to read and write SGF (Smart Game Format) data.
|
This document contains assorted notes about how libsgfc++ presents certain aspects of the SGF standard to the library client.
EBNF terms and their corresponding libsgfc++ types:
ISgfcDocumentISgfcGame, but ISgfcGame objects cannot be nested. Instead an ISgfcGame has a full tree of nodes starting with a root node.ISgfcNodeISgfcPropertyISgfcPropertyISgfcPropertyValueISgfcSinglePropertyValueISgfcComposedPropertyValueISgfcNumberPropertyValueISgfcRealPropertyValueISgfcDoublePropertyValueISgfcColorPropertyValueISgfcSimpleTextPropertyValueISgfcTextPropertyValueISgfcPointPropertyValueISgfcMovePropertyValueISgfcStonePropertyValueThe following has already been stated in the main README and elsewhere, but to reiterate:
ISgfcDocumentReader reads SGF data, the object tree under the resulting ISgfcDocument is already in FF4 format.The EBNF in the SGF standard says that a property must have at least one value. This can be a value that has the value type "None", which the SGF standard defines to be an empty string. The meaning for SGF content in its raw string form is clear: Something like XY[] is possible.
In the libsgfc++ object model, however, a "None" value is not an empty string. Instead it is translated to an empty collection of property values. In other words, if a property has no values the method ISgfcProperty::GetPropertyValues() returns a collection with zero elements.
By default libsgfc++ handles properties that existed in SGF standards before version 4, but were removed from SGF standard 4, as follows:
Library clients that do not want to see pre-FF4 properties can use SgfcArgumentType::DeleteObsoleteProperties when they read in SGF content. This does not prevent the conversion of the properties "L" and "M" to their modern FF4 counterparts "LB" and "MA. Library clients that do not want these conversions to occur must use <tt>SgfcArgumentType::DeletePropertyType</tt> to explicitly delete these properties.
@section autotoc_md85 Property value validation
libsgfc++ makes no attempt to check the validity of property values assigned to a property with, for instance, <tt>ISgfcProperty::SetPropertyValues()</tt>.
A last-ditch attempt at validation is made only when the SGF data is written to a destination, but this is unlikely to catch all mistakes. In any case, it might pay to examine the exit code and messages that are returned as the result of a write operation.
It is ultimately the responsibility of the library client to take care that the values are valid. The consequence of faulty values is that the library client might be unable to read in the game tree again after it has been written out.
@section autotoc_md86 Escaping
@subsubsection autotoc_md87 Summary
Wherever possible libsgfc++ performs escaping and unescaping on behalf of the library client, so that the library client can work naturally with unescaped values at all times.
libsgfc++ does escaping and unescaping for properties of type Text, SimpleText, Move, Point and Stone, but not for properties of the primitive types Number, Real, Color and Double.
<strong>Note:</strong> If the library client performs its own escaping where it is not necessary, the resulting SGF content will be "double-escaped".
@subsubsection autotoc_md88 SimpleText and Text property values
Library clients never need to deal with escaping and unescaping in SimpleText and Text property values.
When the library client programmatically creates an <tt>ISgfcSimpleTextPropertyValue</tt> or an <tt>ISgfcTextPropertyValue</tt> object it can simply provide the natural string value without escape characters. Later libsgfc++ will add the necessary escape characters at the time it generates the SGF content.
When it reads in SGF content libsgfc++ removes all escape characters from SimpleText and Text property values so that the library client is presented with property values that do not contain any escape characters.
@subsubsection autotoc_md89 Move/Point/Stone property values
Move/Point/Stone property values for Go (GM[1]) are simple two-letter strings that do not need escaping.
Move/Point/Stone property values for other game types are different: Because libsgfc++ does not know the format of these values (libsgfc++ only has game-specific knowlege about Go) it cannot make any assumptions regarding escaping except for the most basic rules that are required for the integrity of the SGF format. The library client therefore does not have to deal with the following:
- Escaping/unescaping a closing bracket character ("]").
- Escaping/unescaping a colon character (":") when the value is the first value of a composed value.
All other cases of escaping/unescaping are the responsibility of the library client.
@subsubsection autotoc_md90 Test cases
Run the test file <tt>escaping.sgf</tt> through the example program to see what your library client will get from libsgfc++ in various scenarios.
@iverbatim
./build/example/libsgfcplusplus-example docread test/test-files/escaping.sgf
@endiverbatim
@section autotoc_md91 Line breaks
TODO
@section autotoc_md92 Compressed point lists
@subsubsection autotoc_md93 Game of Go
libsgfc++ has first class support for compressed point lists if the game type is Go. See the corresponding section in the "Go support" section below for details.
@subsubsection autotoc_md94 Other games
If the game type is not Go, however, support for compressed point lists is only very limited, because libsgfc++ does not know how Point values are structured for non-Go games.
When reading SGF content libsgfc++ makes rectangles that it detects in point list property values available as <tt>ISgfcComposedPropertyValue</tt> objects consisting of two <tt>ISgfcPointPropertyValue</tt> objects. libsgfc++ makes no attempt to expand such rectangles, this is the responsibility of the library client.
When writing SGF content libsgfc++ writes out point list property values in the document object tree as-is. libsgfc++ makes no attempt to compress point lists, this is the responsibility of the library client.
The SGF standard specifies that "[...] points have to be unique, i.e. overlap and duplication are forbidden". libsgfc++ makes no attempt at enforcing this restriction, neither when reading nor when writing SGF content.
<strong>Important:</strong> All of the above is also valid for properties that have a "stone list" value type (e.g. "AB"). The only difference is that when it detects a rectangle value in a stone list property libsgfc++ makes it available as <tt>ISgfcComposedPropertyValue</tt> object consisting of two <tt>ISgfcStonePropertyValue</tt> objects. This behaviour of libsgfc++ ist not conforming to the SGF standard and may change in the future.
@subsubsection autotoc_md95 Test cases
Run the test file <tt>compressed-point-list.sgf</tt> through the example program to see what your library client will get from libsgfc++ in various scenarios.
@iverbatim
./build/example/libsgfcplusplus-example docread test/test-files/compressed-point-list.sgf
@endiverbatim
@section autotoc_md96 Go support
@subsubsection autotoc_md97 Go-specific C++ types
The SGF standard does not specify how to interpret Move, Point and Stone property values. However, the library treats Go as a first class citizen, in a similar way that SGFC does, and provides the following Go-specific C++ types:
- <tt>ISgfcGoPoint</tt> and <tt>ISgfcGoPointPropertyValue</tt>
- <tt>ISgfcGoStone</tt> and <tt>ISgfcGoStonePropertyValue</tt>
- <tt>ISgfcGoMove</tt> and <tt>ISgfcGoMovePropertyValue</tt>
In order for this support to be possible, the library needs to analyze the game type and board size properties (GM and SZ). For this purpose the library makes the following special property C++ types available:
- <tt>ISgfcGameTypeProperty</tt>
- <tt>ISgfcBoardSizeProperty</tt>
@subsubsection autotoc_md98 Pass moves for Go games
For Go <a href="https://www.red-bean.com/sgf/go.html#types" >the SGF standard defines</a> that black or white pass moves can have either value "" (an empty string) or "tt". The latter counts as pass move only for board sizes <= 19, for larger boards "tt" is a normal move. The SGF standard also mentions that "tt" is kept only for compatibility with FF3.
libsgfc++ always exposes pass moves as an empty string to the library client. This is unambigous and conforms to FF4. This means that when <tt>ISgfcDocumentReader</tt> reads SGF content with "tt" property values, the resulting <tt>ISgfcDocument</tt> will contain pass moves with empty string property values.
When the library client programmatically sets up an <tt>ISgfcDocument</tt> with pass moves that have value "tt", and the document is then passed to <tt>ISgfcDocumentWriter</tt> for writing, the resulting SGF content will contain pass moves with empty string property values.
@subsubsection autotoc_md99 Compressed point lists
libsgfc++ automatically expands compressed point lists when it reads SGF content and the game type is Go (GM[1]). So when <tt>ISgfcDocumentReader</tt> reads SGF content the resulting object tree will contain one <tt>ISgfcPointPropertyValue</tt> object for each individual point on the Go board, whereas the original SGF content might have contained only a single property value.
When it writes SGF content and the game type is Go (GM[1]) libsgfc++ behaves in one of the following ways:
- By default libsgfc++ automatically compresses expanded point lists, and leaves already compressed point lists alone so they remain compressed.
- When the library client specifies the argument <tt>SgfcArgumentType::ExpandCompressedPointLists</tt>, libsgfc++ automatially expands compressed point lists, and leaves already expanded point lists alone so they remain expanded.
When libsgfc++ expands a compressed point list it normalizes the resulting expanded point list. When libsgfc++ compresses an expanded point list it does so in the most efficient way. This means that in roundtrip processing, when SGF content is first read and then written, it is impossible to preserve the original format in all cases.
<strong>Important:</strong> Since Go does not have a distinguishable set of game pieces (all stones look and behave the same) the SGF standard allows the Stone value type to be the same as the Point value type. As a consequence properties that have a "stone list" value type (e.g. "AB") also support point list compression, and all of the above is also valid for "stone list" properties. The only difference is that when <tt>ISgfcDocumentReader</tt> reads SGF content the resulting object tree contains <tt>ISgfcStonePropertyValue</tt> objects.
@subsubsection autotoc_md100 The "KI" and "KM" properties
According to the SGFC readme document the "KI" property is a private property of the "Smart Game Board" application (SGB). The property name means "integer komi".
libsgfc++ converts "KI" to the Go-specific "KM" property, dividing the original "KI" numeric value by 2 to obtain the new "KM" value. libsgfc++ performs this conversion in all cases, even if the game tree's game type is not Go.
When libsgfc++ reads SGF content it treats the "KM" property as having a Real value when the game tree's game type is Go. For all other game types it treats the "KM" property being a custom property with an unknown property value type.
@section autotoc_md101 Inheritable properties
TODO
@section autotoc_md102 Character encoding
@subsection autotoc_md103 Introduction
The SGF standard specifies the <tt>CA</tt> property which allows an SGF-writing application to specify the character encoding in which a game tree's SimpleText and Text property values are stored.
This approach is somewhat antiquated as it not only allows SGF content to be partially encoded (SimpleText and Text property values are encoded, but not the rest of the content), it even allows different character encodings to be used within the same piece of SGF content! From a modern-day point of view this seems strange, and it definitely breaks the notion of an SGF file being "just a regular text file".
An additional complication to the partial encoding approach are the SGF standard's escaping rules. See the section "Escaping and multi-byte character encodings" below for details.
All things considered it is reasonable to assume that modern-day SGF-writing applications that support non-ASCII SGF content are not strictly conforming to the SGF standard, but are using the common-sense approach of encoding the <strong>entire</strong> SGF content with the same character encoding.
It is recommended that library clients use the UTF-8 character encoding whenever they can, first and foremost because it is a sensible universal encoding, and second because the out-of-the-box behaviour of libsgfc++ (and the backend SGFC) are geared towards using UTF-8. The following sections are considered mandatory reading if you plan to use something other than UTF-8.
@subsection autotoc_md104 Encoding modes
The library client can choose between 3 encoding modes when it reads SGF content with <tt>ISgfcDocumentReader</tt> or writes SGF content with <tt>ISgfcDocumentWriter</tt>. The selection is made by specifying <tt>SgfcArgumentType::EncodingMode</tt> when the read or write operation is performed.
The documentation of <tt>SgfcArgumentType::EncodingMode</tt> has the details about each of the 3 modes. The following is just an excerpt to provide an overview:
- Mode 1 (the default mode): A single encoding is used to decode the <strong>entire</strong> SGF content. The encoding to use is detected by probing the SGF content. If a Unicode BOM marker is found then the encoding defined by that marker is used, but the first CA property in the decoded SGF content must match the encoding detected from the BOM. If no Unicode BOM marker is found then the SGF content is probed for CA properties, and the first CA property value found is used as the encoding. If no CA property is found the default encoding ISO-8859-1 is used. Note that only this mode allows the wide-character encodings UTF-16 and UTF-32 to be processed (via BOM detection).
- Mode 2 (specification conformant): A separate encoding is used to decode each game tree in the SGF content. The encoding is defined by the CA property value of that game tree. If a game tree has no CA property the default encoding ISO-8859-1 is used. Only SimpleText and Text property values are decoded! The SGF formatting skeleton as well as property values that are not SimpleText or Text are parsed using ASCII/ISO-8859-1.
- Mode 3: No decoding takes place.
<strong>Important:</strong> When mode 1 or 2 are used for reading, the SGF content is decoded and made available to the library client as <strong>UTF-8</strong>.
Please make sure to read the documentation of <tt>SgfcArgumentType::EncodingMode</tt> to understand the details. You may also wish to consult the documentation in SGFC's README file.
@subsection autotoc_md105 ASCII-compatible vs. ASCII-safe
Term definitions:
- ASCII-compatible: A character encoding that is ASCII-compatible encodes ASCII characters with their single byte values from the ASCII table. Example: A closing bracket <tt>]</tt> character is encoded as a single byte that has the hex value 0x5d.
- ASCII-safe : A character encoding that is ASCII-safe is also ASCII-compatible, but in addition it guarantees that the byte values of ASCII characters are not used to encode anything else but ASCII characters. Example: The hex byte value 0x5d does not appear anywhere in the encoded byte stream except when it encodes a closing bracket <tt>]</tt> character.
Encoding examples:
@icode
ASCII ASCII
Encoding compatible safe ] \ 因 申
-------------------------------------------------------------------------------------------
UTF-16 No No 0x00 0x5d 0x00 0x5c 0x56 0xe0 0x75 0x33
Big5 Yes No 0x5d 0x5c 0xa6 0x5d 0xa5 0xd3
Shift_JIS Yes No 0x5d 0x5c 0x88 0xf6 0x90 0x5c
UTF-8 Yes Yes 0x5d 0x5c 0xe5 0x9b 0xa0 0xe7 0x94 0xb3
@endicode
Discussion:
- UTF-16 is not ASCII-compatible, because it uses two bytes to represent all characters.
- In the Big5 encoding the 0x5d byte value is used to encode both the closing bracket and the 因 character.
- In the Shift_JIS encoding the 0x5c byte value, which in ASCII represents the backslash character, is used to encode the 申 character.
- UTF-8 is ASCII-safe because the 0x5d and 0x5c values are not used for anything else except to encode the closing bracket and the backslash characters. Note: The table above does not demonstrate this, it is stated as a fact with the knowledge of how UTF-8 works. If in doubt consult the "Layout of UTF-8 byte sequences" table on the <a href="https://en.wikipedia.org/wiki/UTF-8#Encoding" >UTF-8 Wikipedia page</a>.
@subsection autotoc_md106 Escaping and multi-byte character encodings
As demonstrated in the previous section, the Big5-encoded character <tt>因</tt> contains the 0x5d byte value that is also the ASCII value of a closing bracket character <tt>]</tt>. Now let's look at this simple piece of SGF content:
@iverbatim
(;FF[4]CA[Big5]C[因])
@endiverbatim
Taking a hex dump of this SGF content reveals the individual byte values:
@iverbatim
28 3b 46 46 5b 34 5d 43 41 5b 42 69 67 35 5d 43 5b a6 5d 5d 29
( ; F F [ 4 ] C A [ B i g 5 ] C [ 因 ] )
^ ^
@endiverbatim
Note the two byte values 0x5d towards the end. A strictly standard-conforming SGF-writing application would be required to escape the first 0x5d to prevent it from being interpreted as the closing bracket of the <tt>C</tt> property value.
Another example, this time with the <tt>申</tt> character which, as demonstrated in the previous section, when encoded with Shift_JIS contains the 0x5c byte value that is also the ASCII value of the backslash character <tt>\\</tt>. The SGF content
@iverbatim
(;FF[4]CA[SJIS]C[申])
@endiverbatim
looks like this in the hex dump:
@iverbatim
28 3b 46 46 5b 34 5d 43 41 5b 53 4a 49 53 5d 43 5b 90 5c 5d 29
( ; F F [ 4 ] C A [ S J I S ] C [ 申 ] )
^ ^
@endiverbatim
Note the byte values 0x5c towards the end. A strictly standard-conforming SGF-writing application would be required to escape this 0x5c value to prevent it from being interpreted as escaping the closing bracket of the <tt>C</tt> property value.
<strong>Conclusion:</strong> These two examples show how impractical it is to program a strictly standard-conforming SGF-writing application. Such an application would have to operate on two levels: First on a character level to perform the encoding of the human-readable text, then on the underlying byte-stream level to perform the escaping required by the SGF standard.
Furthermore, after the escape-processing the resulting SGF content might not even be valid text anymore, because inserting the escape characters might have invalidated the encoding. The Big5 encoding example above is such a case.
@subsection autotoc_md107 UTF-8 is recommended for programmatically creating ISgfcDocument object trees
A library client that programmatically creates <tt>ISgfcDocument</tt> object trees should use UTF-8 to encode SimpleText and Text property values. UTF-8 is a universal and ASCII-safe character encoding that causes the least amount of trouble along all routes.
Specifically, <tt>ISgfcDocumentWriter</tt> supports only ASCII-safe character encodings to write an <tt>ISgfcDocument</tt> object tree. The escape-processing discussed in the previous section is the root cause for the restriction. Read the <tt>ISgfcDocumentWriter</tt> class documentation for some technical details.
@subsection autotoc_md108 UTF-8 support in <tt>ISgfcDocumentReader</tt>
As mentioned in a previous section, when encoding mode 1 or 2 are used the SGF data is read into memory converted to UTF-8. However, without further measures the document object tree would still contain whatever CA property values were found in the SGF data. If these values are not UTF-8 then the document object tree would now contain wrong information about the character encoding used in-memory. This would be a problem if, for instance, the document object tree were passed to ISgfcDocumentWriter, because the writer would then use the wrong CA property values.
To avoid this situation <tt>ISgfcDocumentReader</tt> performs the following manipulations of the document object tree after the read operation is complete:
- If encoding mode 1 or 2 were used for reading <tt>ISgfcDocumentReader</tt> writes the CA property value "UTF-8" into the root node of each ISgfcGame it finds in the document. Only the root node needs to be treated such because the CA property is a root property which according to the SGF standard is only allowed in the root node. Note that the SGFC backend deletes any CA properties that it finds outside of a root node.
ISgfcDocumentReader does not write a CA property value because the SGF data was read into memory as-is.ISgfcDocumentWriterISgfcDocumentWriter configures itself to use UTF-8 as the default character encoding. A library client that programmatically creates a document object tree from scratch using UTF-8 data therefore does not need to explicitly set a CA property value when it wants to write the document.
Library clients that want different behaviour can follow this guide when working with ISgfcDocumentWriter:
1.10.0