Kang Seonghoon, 2013-11-13
Other languages: 日本語 (外部リンク)
This is a note about possible extensions to BMS. The author intends to implement most of these extensions in Sonorous, the BMS player currently in development, but as always it can be arbitrarily delayed. Feedbacks and suggestions are welcomed; see http://mearie.org/about/contact for contact.
The terminology for the BMS format is based on INTERNALS.md of Angolmois: https://github.com/lifthrasiir/angolmois/blob/master/INTERNALS.md#glossary.
The terminology for requirements, including "MUST (NOT)" and "SHOULD (NOT)", is borrowed from RFC 2119: https://tools.ietf.org/html/rfc2119.
P.S. Okay, I thought this is not necessary but I should have mentioned that: this document is a request for comments and you should regard it as such. (And this is not equal to that RFC, which is just a synonym to the Internet Standard.) If you think a particular proposal is nonsense, hard to use, or hard to implement then please speak out as that is a main reason to release this document. Thank you for your consideration.
Each proposal has one of the following statuses:
The problem is recognized but proposal isn't.
The draft version of the proposal exists but needs much elaboration. Sometimes the proposal itself can be withdrawn.
The version of the proposal that can be readily implemented exists. In this stage, the proposal does not change except for implementor feedbacks.
Sonorous implements this particular proposal. For commands, vendor prefix #SNRS: may be required to use it.
At least three distinct implementations exist.
Status: Implemented as of 2013-11-13
While BMS is originated from Japan, nowadays many BMS files are produced outside Japan and it has become a great pain to use non-Latin characters (and even some Japanese characters) in BMS. It is arguably the most wanted standard in the BMS world.
The BMS file is parsed in the UTF-8 mode when one of the following conditions are met:
EF BB BF, i.e. UTF-8 BOM (U+FEFF); orOtherwise the implementation may use an unspecified procedure to detect the character encoding of the BMS file.
If the implementation supports an indented command (one or more whitespaces followed by a sharp #), then it is also required to support U+FEFF as a whitespace character. Except for this recommendation the implementations can continue treating the BMS file as a mostly-ASCII-only format; in particular, it is not required to support all variety of whitespace characters in Unicode as an argument separator.
UTF-8 in this proposal refers to the encoding specified in RFC 3629. This proposal does not require the implementations to implement more sophisticated Unicode standards (e.g. Unicode normalization algorithm).
The reader MUST parse a BMS file starting with the byte sequence EF BB BF in the UTF-8 mode.
The reader MUST parse a BMS file with the first line (up to the first 0A byte) or the first 1,024 bytes, whichever comes first, being a valid UTF-8 string and an invalid 7-bit ASCII string in the UTF-8 mode.
In the UTF-8 mode, if the reader ignores the preceding letters of the particular kind before the command, U+FEFF SHOULD be included in the ignored letters.
In the UTF-8 mode, the reader SHOULD substitute an invalid byte sequence as U+FFFD "REPLACEMENT CHARACTER"s, which number is unspecified but SHOULD be at least one.
The writer SHOULD start the first line (up to 1,024 bytes before the first 0A byte) as a non-command valid UTF-8 string, which includes the byte sequence EF BB BF. The writer SHOULD NOT use the first line exceeding 1,024 bytes.
The writer SHOULD NOT write an invalid UTF-8 byte sequence anywhere.
The writer SHOULD NOT use non-ASCII strings except for the first line (used for triggering the UTF-8 mode) and for the commands with an unparsed string argument. In particular, the writer SHOULD NOT use non-ASCII strings in the commands with a path argument, e.g. #WAVxx and #BMPxx.
It is best to force UTF-8 in every case, but the deep legacy of BMS format prohibits this choice. Therefore we should declare BMS files with UTF-8 encoding in mind as such, in order to distinguish them from other BMS files with legacy encodings. It is also highly desirable that eventually every new BMS file is encoded in UTF-8 and not in legacy encodings.
ruv-it's #CHARSET UTF-8 proposal (which had been withdrawn later) allows for the use of legacy encodings (EUC-KR and SHIFT-JIS). Also note that there are significant variants of both EUC-KR (CP949 aka UHC) and SHIFT-JIS (JIS X 0213). WHATWG's Encoding standard is much more complete in this regard, but it would be an implementation burden as most of supported encodings will be never used.
Alternatively, we can think of (say) #ISUTF8 directive which is equivalent to aforementioned #CHARSET UTF-8 but without any possibility of extension. Since the implementation should detect such directive before any line with UTF-8 sequence, such directive should be the first command. This kind of restriction is hard to enforce as any other BMS command doesn't work in this way, but more importantly, it is incompatible with several implementations supporting the indentation of BMS commands with full-width whitespace (U+3000). Consequently we should avoid introducing any new command.
On the other hand, UTF-8 BOM sequence is already supported by some implementations and it is very rare or invalid in virtually every legacy encoding. But byte-order mark (U+FEFF) is a whitespace character, so there is a risk that the editor or preprocessing accidentally removes it without a notice. Therefore there is the second option, which makes a UTF-8-triggering sequence obvious: the first line should be a valid UTF-8 string with at least one byte with most significant bit set so that it is definitely not ASCII. (1,024 byte limit is there to avoid the infinite lookahead and otherwise arbitrary.) This was inspired by Unicode snowman character, which was a workaround for a bug in IE8 or earlier. It is possible that some character encoded in legacy encodings may be misinterpreted as UTF-8, but the first line of BMS file is generally not a command (thanks to BMSE) or either #PLAYER command so this risk is small enough to ignore.
There are several more small concerns:
Status: Writing
Currently two-letter alphanumeric keys universally used by various places in BMS are limited to 1,296 entries (36 * 36), which seriously limited the use of complex effects like the BGA overlay and ultimately led the introduction of direct movie files in BMS. In the longer term it is desirable to break this limitation.
Alphanumeric keys can be replaced by enclosed alphanumeric keys, which start with (, contain zero or more letters except ( and ), and end with ). Enclosed alphanumeric keys share the same namespace as alphanumeric keys, which means...
00 is equivalent to () (in the data section) or (00) (in the non-data commands).01 is equivalent to (01).ZZ is equivalent to (ZZ).Enclosed alphanumeric keys are case-insensitive just like alphanumeric keys.
This syntax is applicable to every place which supports the alphanumeric keys, including but not limited to the data section, #EXRANKxx, #TEXTxx, #BPMxx, #STOPxx, #LNOBJ, #CHANGEOPTIONxx, #WAVxx, #BMPxx, #BGAxx, #ARGBxx and so on. In this proposal only the alphanumeric keys after the colon (:) in the data section are applicable.
The syntax allows any non-parentheses letter in the enclosed alphanumeric keys, but any keys besides zero or more alphanumeric letters (0-9, A-Z) are reserved for the future extension. Empty enclosed alphanumeric key () is reserved outside the data section.
The reader MUST be able to parse the valid enclosed alphanumeric keys placed within the intrinsic line length limitation, whether the keys are reserved or not. The reader SHOULD ignore reserved enclosed alphanumeric keys in the data section, making them effectively equivalent to () or 00. The reader SHOULD ignore non-data commands including reserved enclosed alphanumeric keys.
The reader MUST treat enclosed alphanumeric keys as a string with the fixed length, not a base-36 number. Therefore (1), (01) and (001) are distinct keys.
The reader MUST NOT parse the non-terminated enclosed alphanumeric keys, which start with ( but has no enclosing ) in that line (or the line cut by the intrinsic line length limitation). The reader SHOULD ignore lines with such keys.
The reader MUST NOT parse the enclosed alphanumeric keys preceded by the incomplete (i.e. one-letter) alphanumeric key in the data section. For example, #00101:abc(de)f is invalid, although #00101:ab(cd)ef is equivalent to #00101:abcdef. The reader SHOULD ignore lines with such keys.
The reader MAY put a limitation on the total unique enclosed alphanumeric keys per the namespace, but such limitation SHOULD be no smaller than 1,296 keys per the namespace.
The reader MUST NOT ignore the whitespaces around the enclosed alphanumeric keys in the non-data commands. In particular, if #FOOxx 42 is a valid command and #FOO 42 is not, then #FOO(xx) 42 is equivalent to #FOOxx 42 but #FOO (xx) 42 or #FOO(xx)42 is invalid.
The writer SHOULD use the ordinary alphanumeric keys instead of two-letter enclosed alphanumeric keys. The writer SHOULD use an ordinary alphanumeric key, uppercased, in the #LNOBJ command for the compatibility reason.
The writer SHOULD use ordinary alphanumeric keys from 01 to ZZ for first 1,295 entries, or all entries if the number of entries is less than 1,295. The writer MAY arrange the order of entries to have important entries included in the compatible region.
The writer SHOULD NOT use () for the compatibility reason. The placeholder in the data section SHOULD be written as 00 as it had been before.
It is not common that one BMS file requires 1,200 or more sounds. (It was once common that one BMS file requires 1,200 or more images, but the recent trend of movie BGA makes this very rare.) Therefore the effect of this proposal is very limited, and in fact the implementation does not have to support more than 1,296 resources at all.
The main intention of this proposal lies on the future: image-like effects and sound-like effects can be indefinitely used. The prime example of image-like effects is #BGA, as it is not an actual image but can be used like an image. However the number of #BGA is severely limited by alphanumeric key space, and along with other reasons #BGA was never widespread. Several recent extensions of this kind counteract this problem with a separate channel. For example, nanasigroove's #SWBGA extension ideally should have been a sound combined with an image-like effect, but it instead uses a dedicated channel and #SWBGA resources contain the lane to apply the effect in the future (so only one channel is required). This is reasonable but doesn't scale at all: if 7-KEY DP BMS (i.e. 16 lanes) has 100 sets of animated sprites applied to all lanes, it would require 1,600 #SWBGA resources.
This proposal is also a precursor of many possible extensions. For example the enclosed alphanumeric key (04 x=15 y=15) (which is currently invalid due to non-alphanumeric letters) may specify an image at given left-top coordinate. Or (-) (again, which is currently invalid) may mark the end of long notes instead of #LNOBJ. Although the current proposal avoids them for easy implementation, such extension is certainly possible in the framework of this proposal.
The choice of punctuations ( and ) is currently arbitrary (though it makes the look more natural in my opinion), and it seems that DTXMania uses parentheses for separator in #BGA command so it can change.
Status: Proposed
Different BMS players have different judgement rules for long note objects. Some players count them as two notes (at start and at end), some players count them as one note, some players (notably ruv-it) count them as multiple intermediate notes which number is proportional to the length. Since the number of notes greatly affects the judgement rules and scores, the ability to specify intermediate notes is crucial for consistent game play.
The channels 7x and 8x, currently unused at all, are designated tick objects. When the long note object has corresponding tick objects (which may coincide with the endpoints of the long note), these tick objects override the default grading rules of the player and the combo count and gauge is affected only when the tick object is reached.
For example, the following BMS file has one long note object containing six tick objects (including one at the end of the long note).
#00151:AA0000000000BB00
#00171:0033445566778800Since the beginning of the long note does not coincide with a tick object, the combo count does not increase until the first tick object (33) is reached. If the player stops pressing the key between the tick objects 66 and 77, the combo count is reset and only first 4 tick objects count. If the player was able to finish the long note properly, then the combo count should increase by 6.
The actual rule is a bit more elaborated in order to handle tick objects at the beginning or end of the long note:
Note that, the combo count may increase by more than one if multiple tick objects count towards the combo count at once due to the generous grading. This is also why we don't play the key sounds for these multiple tick objects.
The long note should contain at least one tick object. If the long note does not contain any tick objects, the player is free to use their own judgements to add tick objects. It is possible to have a tick object at unusual place though; for example, one can put only one tick object to the beginning of the very long note, which makes the player keep pressing the key without receiving further combo counts.
Tick objects outside the long note are reserved, and any unsupported tick objects should be discarded (not converted to the BGM).
The reader SHOULD have a means to make tick objects relevant to the game play (for example, by increasing a combo count or a bonus score). The reader SHOULD distinguish tick objects visually, whether they coincide with the beginning or the end of the long note or not. The reader MAY weigh combo counts or scores from tick objects to be lower than normal objects.
The reader MUST ignore all tick objects outside the long note. The reader MUST be able to process tick objects positioned in arbitrarily many data commands. The reader MUST be able to process overlapping tick objects (i.e. tick objects positioned in the same position of multiple data commands with same measure and same channel); the reader MUST NOT crash in that case, and the reader SHOULD NOT add more than two tick objects at the same position (it may ignore all but one of coinciding tick objects, for example).
The reader SHOULD have a default tick object policy (e.g. one tick object per quarter notes) to apply it in the absence of any tick objects.
The reader MUST apply all tick objects up to the current position when the beginning of the long note with tick objects is graded and the remainder of the long note is to be graded. The reader SHOULD NOT play any associated key sounds.
The reader MUST apply a tick object reaching the current position when the long note is being graded. The reader MUST NOT apply a tick object reaching the current position when the long note is not being graded as the player failed to keep pressing the key before. The reader MAY have an additional penalty applied to missed tick objects. The reader SHOULD play an associated key sound to the tick object if any.
The reader MUST apply all tick objects up to and including the end of the long note when the end of the long note with tick objects is graded. The reader SHOULD NOT play any associated key sounds. The reader MUST play a key sound associated to the end of the long note if the reader is able to recognize one.
The writer SHOULD NOT write a tick object outside the long note. Otherwise, all standard writer requirements to the data commands are applied.
The main intention of this proposal is obviously to make long notes more relevant to BMS. This trend is also observable in many commercial music video games with long notes, and some games even apply this principle to the gauge decrease (so that missing long notes is much more critical). This proposal concentrates to the long note combo however, as it is enough for making long notes important and unique game systems should be respected.
Channels 7x and 8x are chosen since they are never used in the traditional BMS format (though apparently DTX uses them) and they have a profound effect on the game play and scoring. Originally it was considered that visible objects (in channels 1x and 2x) overlapping with long notes (in channels 5x and 6x) are converted tick objects, but many (if not most) implementations of long notes misbehave on such overlapping notes so it was abandoned.
While it is possible to fix the global rate of long note combo throughout the BMS file, it is too implicit (e.g. the number of notes is hard to calculate) and hard to customize (e.g. should the rate be dependent to BPM or time?). Explicit tick objects make everything obvious and even allows for the optional tick sound.
Status: Researching
...
Status: Writing
The BMS format is not designed for multiple languages, but there are increasing number of examples with multiple versions of the BMS file according to the language. Having separate BMS files per the language is not desirable for various reasons (for example, it completely wrecks the internet ranking via the BMS hash), and in most cases only some assets and strings are required to be localized.
The metadata commands such as #TITLE and asset commands such as #BMP01 may have a alphanumeric key immediately following the command (#TITLEKO and #BMP01KO, for example), which is interpreted as an ISO 639-1 alpha-2 language code. Note that this is distinct from an ISO 3166-1 alpha-2 country code; Korean should be KO instead of KR, Japanese should be JA instead of JP, Chinese should be ZH instead of CN or TW.
The affected commands include but not limited to:
#STAGEFILE, #BANNER, #BACKBMP, #CHARFILE, #BMPxx, #EXBMPxx, #BGAxx, #@BGAxx, #SWBGAxx, #ARGBxx, #VIDEOFILE#WAVxx, #EXWAVxx#TEXTxx, #SONGxx#TITLE, #SUBTITLE, #ARTIST, #SUBARTIST, #MAKER, #GENRE, #COMMENTBasically, every command that does not affect the actual chart can be localized. This approach allows for the efficient handling of localized BMS files, since there is only one chart (after randomized) with differing assets.
The alphanumeric key for the language should not be preceded by whitespaces, and should be followed by whitespaces. When applied to the asset commands (e.g. #BMP01 a.bmp), the language should be appended after the target alphanumeric key (e.g. #BMP01KO a.bmp).
This proposal, while not necessary, recommends the enclosed alphanumeric keys for the support of some major target languages (e.g. in order to separate zh-tw from zh-cn). In this case the ordinary alphanumeric key is equivalent to the corresponding enclosed alphanumeric key, and it is interpreted as an IETF language tag which is a superset of ISO 639 language code.
The reader SHOULD have a preferred language, and the preferred language MAY be configurable by the user. The reader MAY have multiple preferred languages ordered by the relative preference.
The reader SHOULD ...
The reader with an enclosed alphanumeric key support SHOULD be able to parse a restricted subset of IETF language tags, namely a language code (ISO 639-1 alpha-2) optionally followed by a hyphen and a country code (ISO 3166-1 alpha-2).
The reader MUST be able to match the language or the language tag exactly (in the case-insensitive manner). If the reader supports language tags, the reader SHOULD additionally match language tags strictly specific than the preferred language (e.g. KO-KR in the BMS, KO preferred).
...
Status: Proposed
Historically the BGA canvas of the BMS was limited to 256 by 256 pixels. This is proved to be insufficient in many cases however, and many players have lifted this limitation for the larger screen. Still there are some unresolved ambiguities remain.
A command for explicit canvas dimension, #CANVASSIZE, is introduced. It has two arguments, one for width and one for height. Both the width and the height should be positive.
The specified canvas dimension replaces the legacy limitation on the BGA (256 by 256) to given value. This also changes several places in the BMS with the implicit 256 by 256 dimension: The BGA smaller than the canvas dimension is padded to the transparent black as usual, and the #BGA command crops at the canvas dimension.
The reader MUST recognize a command with the form of #CANVASSIZE <w> <h>, where the space represents one or more whitespace characters recognized, and <w> and <h> represents one to four decimal digits. The reader SHOULD ignore a #CANVASSIZE command with an invalid syntax. The reader MUST use the last valid #CANVASSIZE as a canvas dimension if any. The reader MAY assume an implicit #CANVASSIZE 256 256 in the front of the BMS file.
In the presence of #CANVASSIZE command, the reader MUST stretch (or do the internal calculation that simulates the stretching of) the internal canvas to the specified canvas dimension. The reader SHOULD pad or crop (or do the internal calculation that simulates the padding and cropping of) the image files used as a BGA to the specified canvas dimension, where the original image anchors at the left top corner. When the padding or cropping occurs, the padded region SHOULD be filled to the transparent black. The reader SHOULD stretch the movie files used as a BGA to the specified canvas dimension if any. The reader MAY rescale the resulting image to the intrinsic BGA region when the canvas dimension exceeds the BGA region. When such rescaling occurs, the reader SHOULD keep the aspect ratio of the canvas dimension. (This is to prohibit the malicious BMS file crashing the BMS player; this proposal does not require the BMS players to allocate more memory than the original image files, since cropping and stretching can be simulated by having a pointer to the original image and cropping or stretching information.)
In the presence of #CANVASSIZE command, commands affected by the canvas dimension, including but not limited to #BGA, #@BGA and (legacy) #ExtChr, SHOULD respect the specified canvas dimension. In particular, #BGA and #@BGA commands SHOULD crop given coordinates to the specified canvas dimension.
The writer SHOULD write a #CANVASSIZE command if the BGA exists, even when the canvas dimension is 256 by 256 pixels.
Status: Writing
There are lots of formats derived from BMS, some of them are major enough but others are minor and normally have only one implementation supporting it. Still many of them share the same game play and only differ from the order and appearance of lanes. In the vein of recent #OPTION extensions, it would be useful to explicitly specify the order and appearance of lanes expected by given BMS file.
This proposal adds one command, #LANES, which specifies the lane-channel association and the order and appearance of lanes. It has the following syntax:
#LANES <channel> <spec> <channel> <spec> ...<channel> is a two-letter alphanumeric key for visible object channels (10 to 2Z), and <spec> is a string without any whitespace that describes that lane.<channel> can also be 00, which means that there are out-of-lane specifications. (This also prohibits channel 00 from being resurrected.)
<spec> is further delimited by _ letters, where the first part delimited by _ is mandatory and the following parts gives a hint to the first part. For example, key_blue_even contains one mandatory specification key and two hints blue and even. If the mandatory specification is ignored associated hints should also be ignored. <spec> is an opaque string besides from _, but it is recommended for implementations to use a vendor prefix ending with : to minimize the possibility of collision.
A lane specification is a concise description of recommended lane appearance and behavior. Since it is not mandatory, implementations may completely ignore this command, but it is recommended that i) all objects outside channels appeared in #LANES command are converted to the BGM and ii) the order of lanes is retained.
This proposal also defines the following standard lane and out-of-lane specifications. Implementations are free to ignore them but nevertheless recommended to follow them.
keyA generic on/off button. Has the following standard hints:
oddA key in odd position. Mimicking the appearance of musical keyboards, lanes for such keys are not intended to overlap.
evenA key in even position. It is possible that lanes for even-position keys overlap with odd-position keys, but not with themselves. (For example, if there are 7 keys with odd, even, odd, odd, even, even, odd hints, the second even key may overlap with the first and third odd keys, but the fifth and sixth even keys do not overlap with each other.)
scratchA generic scratch with both clockwise and counter-clockwise motion. Intended to have a bigger visual cue than keys.
pedalA generic pedal. Intended to have a bigger visual cue than keys.
gap (out-of-lane)A gap between lanes. The size of the gap is implementation-dependent and can be ignored. Mainly used as a visual cue for two sets of lanes in Double Play mode.
The following are examples of #LANES command with the existing game modes. (Note that indentations are not a part of the specification.)
the original BMS
#LANES 11 key_odd 12 key_even 13 key_odd 14 key_even 15 key_odd 16 scratch
the original BMS with an explicit free zone (vendor prefix example)
#LANES 11 key_odd 12 key_even 13 key_odd 14 key_even 15 key_odd 16 scratch
17 bm98:free-zone
14-key BME
#LANES 16 scratch 11 key_odd 12 key_even 13 key_odd 14 key_even 15 key_odd
18 key_even 19 key_odd 00 gap 21 key_odd 22 key_even 23 key_odd
24 key_even 25 key_odd 28 key_even 29 key_odd 26 scratch
OCT/FP (simulates #OCT/FP command)
#LANES 21 pedal 16 scratch 11 key_odd 12 key_even 13 key_odd 14 key_even
15 key_odd 18 key_even 19 key_odd 22 key_even 23 key_odd 24 key_even
25 key_odd 28 key_even 29 key_odd 26 scratch
PMS (with multiple hints, some of them not in the standard)
#LANES 11 key_odd_white 12 key_even_yellow 13 key_odd_green 14 key_even_blue
15 key_odd_red 22 key_even_blue 23 key_odd_green 24 key_even_yellow
25 key_odd_whiteSince the lane specification greatly reduces the role of #PLAYER command, #LANES command always overrides #PLAYER command.
...
...
See http://hitkey.nekokan.dyndns.info/cmds.htm for the detailed information about existing channels.
00 [Reserved]
01 BGM
02 Measure length, has a distinct syntax from others
03 Sets BPM, coarser granularity
04 Sets BGA, 1st layer
05 [Reserved; had been used for sprite customization and #SEEK command]
06 Sets POOR BGA
07 Sets BGA, 2nd layer
08 Sets BPM, finer granularity
09 Stops the chart for specified period
0A Sets BGA, 3rd layer
0B-0E (nanasi) Sets the opacity of BGAs
0F-0Z [Reserved]
10-1Z
: Visible objects, also include long notes with #LNOBJ command
20-2Z
30-3Z
: Invisible objects
40-4Z
50-5Z
: Long note endpoints
60-6Z
70-7Z
: (sonorous?) Tick objects
80-8Z
90-96 [Reserved]
97 (fgt) Sets BGM volume
98 (fgt) Sets key sound volume
99 Sets the accompanied text
9A-9Z [Reserved]
A0 Sets the judgement difficulty
A1-A4 (nanasi) Sets the color key of BGAs
A5 (nanasi) Sets the BGA animation reacting to the key presses
A6 (nanasi) Sets the player-specific option
A7-AZ [Reserved]
B0-BZ
: (technicalgroove?) Package?
C0-CZ
D0-DZ
: Bombs with the amount of gauge decreases
E0-EZ
F0-FZ [Reserved]
G0 (technicalgroove internal?) Measure line
G1-G8 [Reserved]
G9 (technicalgroove internal?) Deactivates the ongoing STOP command?
GA-GZ [Reserved]
H0-HZ
: (technicalgroove internal?) Long note endpoint key sounds
I0-IZ
J0-JZ
: [Reserved]
Z0-ZZVendor keys:
(nanasi): nanasigroove(fgt): forgetalia(technicalgroove?): Technical Groove, not implemented yet(technicalgroove internal?): Technical Groove internal use, not implemented yet(sonorous?): Sonorous, not implemented yet