Version 0.1 alpha. 2012.*
**** This document is the draft of V0.1. Everything is subject to change. *****
Cbm8m is a container format. It encapsulates native Commodore music program and data binary dumps, adding complementary technical and administrative information.
Cbm8m can be thought of as an expansion of well established ideas and practices of the High Voltage SID Collection, together with the PSID fileformat in particular, albeit in the form of a completely new structure.
The format builds around three, general principles. It should be an uniform 8-bit music archive format, as far as system similarities and differences permit. Archive files should be playable on the tunes' respective original platforms as far as system resources allow. And last, the format should provide some means to ease the administrative work of 8-bit Commodore music archival projects.
As to what the files are supposed to store, the definition strongly relies on HVSC's established practices.
Cbm8m files use the c8m file extension.
What comes here is a general list of information fields with types and descriptions, grouped by hierarchy. (How they're stored in files, is described separately).
Identifies this filetype. Always required.
This is the version identifier of the file format of some particular cbm8m file. The version field uniquely identifies cbm8m file structure ie. the set of useable fields and the physical layout of the corresponding file. Always required.
File header structure (ie. the very first part of the file, to be used to identify filetype and version) is intended to be kept unchanged amongst all cbm8m structure revisions.
Structural compatibility amongst minor versions of the same major version is guaranteed (semantic changes are possible). Structural compatibility between major versions is preferred, but not guaranteed.
A maximum of 16*16 (major/minor) revisions can be issued. Version numbers are allocated in an ascending, but not neccessarily continuous order.
New revisions should be defined so that files of older versions can be converted to new ones by automated methods / minimal human intervention (as much as possible).
The platform the tune is intended to be played on. This is an enumeration (an index that selects a platform from a pre-defined global dictionary of platforms).
File structure is not affected by platform setting, but the functions (and, consequently, the names) of flags are.
The platform dictionary is supposed to be extended, and platform bytes are, thus, allocated, in the order that the respective platforms get supported. At the time of writing this document, and as per the very first apparent application of this format, only 264 series (C-16, C116, Plus/4) and its respective option flags are defined.
Global settings for the target environment. A maximum of 8 global flags can be used for each type of known platforms.
At this moment, only a single flag is defined: PAL/NTSC (the player environment's required type). That is to say, players need to emulate/initialize a corresponding type of machine for replay, calculations (those that depend on machine type) need to take account of this setting, and so on.
In the case of the 264 version, the default corresponds to PAL setting (NTSC can be specified).
Each cbm8m archive may contain a minimum of 1 and a maximum of 256 subtunes. Each subtunes are described by associated definition blocks.
The subtune which is intended to play by default, after loading the file.
A cbm8m file can contain a minimum of 1 and a maximum of 256 Commodore binary dumps. Each binaries are described by respective definition blocks (defining environment flags, entry point address(es) and so on).
Subtune definitions and binary definitions, together, provide a way to link and map tunes of multifile programs together as a single series of subtunes, into a single archive file, without the need of hacking tunes on the binary code's level. This is supposed to help both tune rippers and native music players (that is to say, neither hacking dumps in machine code, nor dealing with huge binary images that ruin themselves / and more upon initialization is needed anymore).
The title of the music.
The Author(s) of the music.
The release year and publisher of the music.
An optional comment field.
For precise definitions of these fields, and until expanding this document with that, please refer to the HVSC's respective documents.
Subtunes are numbered from 0 to [NumberOfSubtunes-1]. Each subtunes are defined by a set of fields described below.
Subtunes don't have any (hardware- or environment specific) flags.
The init parameter with which the code in the selected binary dump needs to be initialized with, in order to play this subtune. This parameter is loaded to AC (by the player environment) prior to calling the init entry of the respective binary code.
This param defaults to what is expected when there's no specifiable init parameter at all (ie. init parameter <=> subtune index). Otherwise, this parameter can be used to reorder subtunes, skip non-existing (muted, or otherwise erroneous) "subtunes", map subtunes of multiple binary dumps to a single line of subtunes and so on.
Tells where (which binary) the current subtune is ie. which binary needs to be loaded and initialized. Defaults to 0. (A "don't care" parameter unless multiple binaries need to be dealt with).
The subtune's length (elapsed time) in frames, until the subtune loops or ends.
By default (0), subtune length is unknown. Otherwise, the player should play the tune for this number of frames. (As to which way it's best to stop playing - cut, fadeout, cross-fade, just to name some obvious options - is left to the players consideration).
The time measure is affected by both platform setting and option flags (different base clocks and framerates amongst different Commodore machines and television standards). The frame is selected as timebase so that timing can be inherently in sync with the playing environment / the tune.
This field is obviously not intended to be calculated by hand. The idea admittedly comes from HVSC's Song Length Database, which is generated by automatism. This field merely provides the possibility to store such information, per subtune, in this structure.
Original compositions name, if the (sub)tune is a cover of a known, previous musical composition.
The Author(s) of this specific subtune, if different from the global Author setting.
The optional title of this subtune (if it has one).
Used to describe, optionally, the covered compositions Author.
Optional comment.
Similarly to the global textfields, please refer to HVSC's guidelines for these fields usage.
Binary dumps are numbered from 0 to [NumberOfBinaries-1]. Each one of the binaries are defined by a set of fields described below, plus the respective (in the file, separately stored) binary dump.
Unless zero, this is the init entry address of the player code contained by this binary.
If zero, init address is the start address (in memory) of this binary dump.
Prior to calling this init address (in order to init a particular subtune), the player environment is expected to load the respective subtune init parameter into AC.
Unless zero, this is the play entry address of the player code.
If zero, the init routine is expected to install its own IRQ routine (and set up / arrange regular updates on its own).
Length of binary dump in bytes.
In practice, a valid binary dump should never be shorter than 2 bytes. This is not enforced by the fileformat, yet, it must be validated for by program code.
In the textual format, this field is redundant.
In the binary format, leaving this field default (0) indicates that the respective binary dump data must be read until EOF. Obviously, this only makes sense if the respective binary dump is the last part of the file. 99% of files will presumably store a single binary dump per cbm8m file (thus, the possibility here to eliminate some redundancy). As this is an exception to the schema of default values in this structure, additional logics and care is needed for its implementation.
These flags sign either special hardware requirements and special settings of, or with regard to, subtunes contained by this binary dump.
Binary option flags (as well as global option flags) are specific to file version and platform setting.
As of V0.1, and for the 264 series, 4 flags are defined, in this order.
Flags always default to 0 ie. false. That is, for Plus/4, the defaults are ROM, noSID, noDigiBlaster, noBasic (respectively).
If field is zero (both params), then we don't know anything about free memory.
Setting this parameter helps players (especially, player environments expected to run on Commodore machines) to determine the availability of useable ram pages (that is, to be used by the environment itself).
Both parameters are measured in 256-byte long pages. They define the largest useable RAM block, neither occupied, nor accessed by music player code of this binary, by start (first byte) and length (second parameter).
Similarly to the subtune length parameter, this field is intended to be generated by automatism. This process is also required to take option flags of respective program dumps into account. Mapped out or non-existing RAM pages (as per memory config set by option flags) should never be considered "free" in this fields context, even if they're (truly) never occupied or accessed by music player code or data.
Logically, dumps would just belong to binary definition blocks. For practical reasons (such files would be pretty uncomfortable to be dealt with) they're stored by the format separately.
Binary dumps embedded by cbm8m files are uniformly in the usual "data with start address" format. Consequently, valid dumps are required to be at least 2 bytes of length.
Particular field data types reflect binary representation constraints. (Technically speaking, there'd be no need for such constraints if only textual representation was needed).
There may be implied constraints for some of the fields. For example, the number of the default subtune should never exceed the number of subtunes. A valid cbm8m file needs to contain at least one valid binary dump (and so on). Programs capable of saving cbm8m files need to validate data for these cases and warn user about possible inconsistencies upon saving files. (Players are also suggested to do a couple of simple sanity checks before passing control to some binary code loaded from a file).
Character encoding is UTF-8. Line termination can be either, or any combination of CR / LF (all valid).
Leading and trailing whitespaces are ignored. Whitespaces between field names, field numbers, separator characters, and field values are allowed, and ignored. (The file header is an exception to this rule).
Empty lines are ignored.
Lines with leading hash marks (#) are comments (ignored).
Upon reading and interpreting text files, all identifiers are case insensitive.
The file consists of
with the definition blocks being
Consists of an asterisk, a file magic string (CBM8M), the fileformat version, and the platform identifier. Magic, version, and platform are separated by whitespace(s).
The header needs to be the very first line of the file. Leading whitespaces are not allowed.
Example:
*CBM8M V0.1 264
Blocks start with a block identifier, which is followed by a list of field definitions. Blocks extend to the next valid block identifier, or EOF.
Block identifiers generally consist of an identifier name and a colon. Additionally, subtune, binary and binary data block identifiers must also contain an identifier number between the name and the colon. Valid identifier names are Tune, Subtune, Binary, and BinaryData. Valid identifier numbers range from 0 to 255 (decimal).
Examples:
Tune:
Subtune 0:
The global tune definition block must be the very first block following the file header line. The rest of the blocks can follow in an arbitrary order.
Defining some blocks (except for the global one) multiple times is technically valid. Later definitions take precedence. The end result (after having read and interpreted the whole file) is required to be consistent.
Field definitions consist of a field identifier name, followed by a colon, and the field value.
For valid field names, please refer to the "Field definitions" section of this document. Note that field names are contextual.
Fields definitions can be listed in an arbitrary order. Re-definitions are valid, later definitions take precedence.
Numeric values can be defined either in decimal, or in hexadecimal (the later denoted by a leading $ sign). Flags can be specified by assigned names, or (if more apply) a list of assigned names, separated by commas. Text fields are single line strings, without enclosing quotation marks, and without line wrap.
Undefined fields are assumed to be of default values, unless definition is strictly required.
Examples:
Title: After Burner II
Author: Hiroshi Miyauchi (Hiro)
DefaultSubtune: $03
OptionFlags: NTSC
BinaryData is a special definition block; that is, it has no named fields but a single hexadecimal dump of the respective Commodore binary file. No hex prefix/suffix is used. No checksums. Number of bytes per line is arbitrary.
Example:
BinaryData 0:
de 17 85 fa 4c e6 17 4c e5 18 a2 00 8a 9d d2 27
e8 e0 20 d0 f8
The text version is intended to be an aid for tune rippers, music archive project administrators etc., so that files can be edited, stored, organized, be kept track of in a more convenient and predictable way.
Music players may, but are not required to read text files.
The implementation of reading cbm8m structures is intended to be sequential. Context is sometimes set by earlier parts of the file (ie. file version, platform, flags etc.). Later definitions are assumed to take precedence over earlier definitions of the same fields. The parser should be robust, yet forgiving to creative uses and/or general sloppiness of source files (as far as syntax is concerned, and as far as some constructs fit syntax, at all).
Cbm8m files can be identified just by reading and checking the first 6 bytes for the file magic ID (case insensitive *CBM8M for text, CBM8M for binary).
Reading dumps from text files should take the real length of dumps as basis. Consequently, the Len field of binary dump definition blocks is redundant in the textual format.
Processing data should always be performed on internal data structures. The suggested method of processing is "initialize an internal cbm8m structure with defaults - do modifications and additions specified by a source file - do modifications specified by user - do sanity checks - generate output snapshot from the internally stored structure in the required destination format". As a side-effect, special formatting of input files are never preserved, only effective data is.
Generating output is suggested to work on the basis of the "lazy man's practice"; that is, generate output only for 1.) what's strictly required by format spec, 2.) data which differs from default values. This results in smaller, and what's more important, well readable files, because only significant information is listed. Optionally, generators may produce files with fully populated structures and/or such files with the redundant data rows commented, upon the user's request.
Text files produced by cbm8m file generators should be formatted according to a handful of (slightly strict) rules.
The binary format is optimized for size. To achieve this, most fields are optional, defaulting to some purposely selected values, and some fields (strings) are of dynamic size. On the con-side of that, positions of fields are dynamic ie. generally unknown (except for the file header and a few global fields). Consequently, the data needs to be processed sequentially.
All numeric fields are unsigned integers. All multibyte integers are little endian. Numeric fields that are interpreted "1-based" are indicated in the list below by "+1". All strings, unless indicated otherwise, are UTF-8 encoded null-terminated strings. Flags in flag bytes are allocated from bit 0 and upwards.
The order of blocks per file, the order of fields per blocks is fixed.
All fields before and including the respective DefinedFieldFlags field are static (unconditionally present). All fields following DefinedFieldFlags up to and including the last field of the respective block are optional.
Bits of DefinedFieldFlags are assigned to subsequent optional fields (bit 0 --> first subsequent optional field and so on), indicating these fields absence or presence. If 0, the value of the respective field must be considered default, and, with respect to file data, absent. 1 indicates the presence of the respective field.
For field names and descriptions, please refer to the "Field definitions" section of this document.
Offs Field name Len Type Default Comment
0 FileMagic 5 string[5] "CBM8M"
5 FileVersion 1 byte 0 Version major/minor on high/low nybble
6 FilePlatform 1 byte 0 Enum
7 DefinedFieldFlags 1 byte 0 See def.
above
8 OptionFlags 1 byte 0 Global option flags,
packed
?
NumberOfSubtunes 1 byte +1 0
?
DefaultSubtune 1 byte 0
?
NumberOfBinaries 1 byte +1 0
?
Title ?
string "<?>"
?
Author ?
string "<?>"
?
Released ?
string "19??
<?>"
?
Comment ?
string ""
Subtunes are numbered from 0 to NumberOfSubtunes-1. For each subtunes, one subtune definition block is inserted, in an ascending sequential order.
Offs Field name Len Type Default Comment
0 DefinedFieldFlags 1 byte 0 See def.
above
1 BinaryNumber 1 byte 0
?
InitParam 1 byte #subtune
?
SubtuneLength 4 dword 0
?
Name ?
string ""
?
Author ?
string ""
?
Title ?
string ""
?
Artist ?
string ""
?
Comment ?
string ""
Binaries are numbered from 0 to NumberOfBinaries-1. For each binary dumps, one binary dump definition block is inserted, in an ascending sequential order.
Offs Field name Len Type Default Comment
0 DefinedFieldFlags 1 byte 0 See def.
above
1 OptionFlags 1 byte 0 Binary option flags,
packed
?
Len 2 word 0
?
InitAddress 2 word 0
?
PlayAddress 2 word 0
?
FreeMemArea 2 word 0 start: lo,
len: hi
Original, unmodified Commodore binary file dumps (NumberOfBinaries pieces).
Offs Field name Len Type Default Comment
0 StartAddress 2 word - Stard address in memory
2 Data Len[n]* - - Data
*or until EOF
As suggested earlier, these files need to be interpreted sequentially.
There's one important point to note, and this is the distinction between a cbm8m structure (which, ultimately, describes a series of subtunes) and its representation by some particular filetype. The plain information structure, by itself, is meant to be worked with within programs. Files, whichever the type, are merely meant to store this structure, the actual filetype being optimized to some particular conditions. As a general rule, when implementing cbm8m replay, cbm8m file manipulation, anything, always think of working on fully populated internal data structures, and deal with default values, optional fields and such peculiarities solely in the designated loader and saver part. (Note that there still are special field values to be taken into account by, say, player programs - but that's a completely different story. The point is, a cbm8m player, which uses unpacked structures internally, does need to know that, say, the music code installs its own IRQ handler (one special value of a field), but already doesn't need to know that the field in question once did have an assigned default value, let alone that this field was optional in the fileformat that it was loaded from.)
Loading cbm8m binary files may appear complicated at first sight (which it is, but only relatively speaking). The trick actually also lies in the previous point: simply assume that the structure to be loaded is built up from unconditionally present and static sized fields, with the exception that some of these fields were skipped at the time of storing the data into the file. To interpret / load such structure, an algorithm like this can probably be used:
Back then, limited system resources must have meant major concerns behind some specialities of the original PSID file format. Some of these specialities became drawbacks later that had to be worked around, and brought about the evolution of the format (see v2, v2ng and RSID for details). Today, system resources aren't such major concerns anymore. Cbm8m does expect more from a player environment, and in turn, doesn't have to deal with special cases that PSID and its evolutions did. On the con-side, it does imply a couple of special cases because of supported methods of playing tunes (to support simpler tune ripping work and native player helpers).
Generally speaking, the environment, where Commodore 8-bit program code is supposed to run, is very close to the original computers parameters. Level of compatibility might and probably will vary from implementation to implementation. The list below is a general, minimal set of requirements.
To load and play a tune,
Running administrative tasks should be relatively simple in emulator based players. Tune replay and administrative tasks generally don't share the same environment, administration can rather work as a supervisor process above the emulated machine. That way, the loaded Commodore code is free to do basically anything in the emulated environment, without the risk of interfering with player interface and control tasks.
One thing presumably very hard to do on an even bearably working level, is to build a player environment that has to cooperate with basically arbitrary music player code, with considerable resource constraints as an extra handicap. Yet, given that the real thing is the only environment of 100% approximation of itself, as far as legacy music is concerned, such option must be supported. Cbm8m has the intent of supporting such players (not that it'd make that task simple, but sorts cards so that a couple of things would just work out of the box, and other things become simpler or merely possible to do).
Some features that supposedly help creating such programs:
There are four different cases of how some particular tunes need to be initialized and played.
The basic form of playing tune rips. Probably more than 90% of tunes just fall into this category. This needs a tune play address to be specified.
First, tune init needs to be called (subtune index loaded to AC). Tune init address is either specified directly, or left 0, which merely means that the binary's start address also plays the tune init address' role (either way, tune init can be called). Optionally, tune init may modify any registers, memory addresses, and especially the interrupt flag. However, the routine is explicitely required to return to the caller (as well as the tune play routine), and mustn't install any IRQ handler. After control is returned, the player environment needs to call the tune play address repeatedly upon any valid / enabled asserted hardware IRQs, and clear the respective interrupt flags each time afterwards. (Installing an IRQ handler might be optimal. When doing that, memory mapping needs to be taken into account). The environment should also measure time and update the screen (presumably doing that in the foreground).
(One moral of the story, that although most tunes play from a single raster interrupt, so, player init is unlikely to mess with IRQ masks and timer settings by itself, optionally, it is legal to do that. Tunes may play updated by timer interrupts, tune play routines may mess with the raster compare register, this is all completely legal. In this method, only one thing is illegal: not giving control back to the player environment.)
If play address is unspecified (0), that means that the tune installs its own IRQ handler. There are yet two options for that case. The reason is that init isn't required to return control to the player environment (so that busylooping could be supported).
For this moment, let's suppose that the tune init routine does return after having installed the IRQ handler. From that on, this is actually pretty easy. Everything is just set up by player init, the environment only has to enable interrupts (tune init may, but not required to do that) and let the show run. Of course, environment has to do its own work of updating time and screen, handling keyboard etc. in the foreground, but otherwise everything is set up and done by tune code.
Tunes sometimes need to do busy looping in order to create special sound effects and such (Galway's noise on C64 as typical example). In that case, control can't be given back to the environment.
For a native player environment, it's almost impossible to cooperate with that (given no CPU protected mode / supervisor runlevel). Some hack must be done, in order to make cooperation plain possible. In cbm8m's respect, this is a kind of "user exit" that the player environment is assumed to set up prior to initializing tunes, and the ripped tune is expected to call from time to time, so that the environment can do its own administration work and return control to the tune.
(Remember, we're talking about a presumably very small percent of tunes).
There are two things to be implemented - verify if code was granted control by a cbm8m environment (remember, one goal of such archeology projects is to preserve stuff, not just merely producing files that can be played under some particular environments, so the event of someone extracting tune binaries and start playing around with them, without any cbm8m player to set up environment, is completely predictable). If it was, set up user exit.
In cbm8m, parameters are passed on the stack. The player environment is expected to push three bytes on the stack prior to calling init. Init is expected to read these bytes. (It can probably just pull bytes and "throw them away"; it likely won't attempt to return to the caller anyway). Return address aside, the first two bytes (in the order of pulling them off the stack) hold the address of the user exit (first: lo, second: hi). The sum of all 3 bytes need to give $AA (so that the validity of the user exit address can be verified).
The frequency of calling the user exit is expected to range from some ten rasterlines to some frames. The player environment is expected to keep track of time by itself, regardless of the actual frequency of getting in control.
Basic tunes are denoted by an option flag of the tune binary dump.
For playing Basic tunes, players must do a couple of things differently. Both the init and play addresses become redundant. Tunes are started by executing a Basic ROM Run command and are left running on their own.
Subtune init parameter is passed on to the code by setting up an USR() function handler by the player environment. USR is a standard Basic function amongst all Commodore Basic flavours. Basic code first has to determine whether it runs under a cbm8m player (by reading the USR vector checking whether it's platform default or redefined). Then get subtune index by executing USR() (parameter is arbitrary). This is supposedly a one or two lines addition to Basic code (ripped code can be kept clean).
Playing a tune that has been coded in Basic is technically the same as playing a busy looping tune. However, there's one significant difference: the code predictably never messes with IRQ settings, system memory area, or memory mapping. That allows the player environment to attach its own code to the Kernal IRQ chain and get in control repeatedly. Consequently, playing Basic tunes needs no calling of user exits or any similar tricks.
For native players, playing subtunes of a cbm8m archive might impose problems. Reloading the archive every time a new subtune is selected might turn out just overkill (and very slow). The player environment should implement a list of reasonable initializations, and attempt to init another subtune, in case the old and new subtunes are to be found in the same binary dump (which is currently in memory). For emulation based players, re-initializing the complete environment should not be a problem.
There are usually restrictions as to what some particular code, stored in an archive format, is allowed and forbid to do. In cbm8m, the code may basically do anything, load to any addresses, copy its portions to any addresses and so on. The consequence is, that rips never need to be refactored for just an archive systems' needs, with the conside, that rips either work or don't work in some particular environments. Compatibility with respect to emulation based players should still never be a problem (see notes on supervision), native players may suffer. There are a few guidelines, however, that should help that percentage of working rips to rise to a considerably high level.
Cbm8m is (C) 2012 Levente Hársfalvi. This document is distributed under the Creative Commons by-sa (Attribute-ShareAlike) license ( http://creativecommons.org/licenses/by-sa/3.0/ ). In short, you can adapt, share, make commercial use of this work, as long as it is attributed, and modified works are distributed alike.
This is originally written in perl's pod format and you can find it in html or text (or maybe even LaTex).
Current version of this document should always be found at http://www.harsfalvi.net/cbm8m .
The author can be contacted here: public mailbox at harsfalvi dot net. (No spaces, replace 'at' and 'dot').
The predecessor of cbm8m, which should be considered a temporary, experimental, single-platform format, came around at some point of a discussion and series of experiments concerning the presumable foundations of a Commodore Plus/4 music archeology project, with its respective fileformat and emulation-based music player system. It was Attila Grósz, author of TedPlay, who implemented the early versions of the format in his player, providing valuable feedback and suggestions. It was also Attila who, still in the very early stages, suggested to rather go for a multiplatform format.
The PSID file format (v1, v2) has been created by Michael Schwendt. PSID extensions (PSID v2NG, RSID) have been developed by Dag Lem and Simon White.
The SID Tune Information List (STIL) is (C) by the HVSC Crew.
The SID Song Length Database has been invented and implemented by Michael Schwendt.