inform7/notes/working_notes.md

# Ongoing work: a few notes

This rudimentary blog-like page is to provide brief notes on changes to the
Inform code base made between formal releases, and is only likely to be useful
to people experimenting with unreleased builds of Inform.

## Current status

Releases on the `10.1` branch are now available, and in principle we can begin
work on new material for 10.2. Any urgent bug fixes will be made with point
releases on 10.1: we're currently at 10.1.2.

## News items

### Breaking ground on IE-0001 (7-10 December 2022)

Implementations are now mainly in place for the first round of evolution proposals,
with the conspicuous exception of
[IE-0001 Directory format for extensions with resources](https://github.com/ganelson/inform-evolution/blob/main/proposals/0001-extensions-with-resources.md).
This requires significant updates to the build manager, and belongs to the
measure-twice-cut-once category of changes: IE-0001 will likely be heavily
revised as we go along. Still, eventually it's time to implement, and the first
facilitating work is done.

An extension `Sample` by `Example User` can now be stored as the directory:

	Example User
		Sample
			extension-metadata.json
			Source
				Sample.i7x

instead of the single file

	Example User
		Sample.i7x

The extension metadata is required to give title, authorship and version data
exactly matching that given by the extension file itself, so if the extension
opens with

	Version 2 of Sample by Example User begins here.

...then the JSON metadata file must contain exactly this `is` object:

	{
		"is": {
			"type": "extension",
			"title": "Sample",
			"author": "Example User",
			"version": "2"
		}
	}

The build manager now has a new `inbuild_genre`, called `extension_bundle_genre`,
to represent extensions stored in this way; thus, just as Inform projects have
two different genres, `project_bundle_genre` and `project_file_genre`, so
extensions have also. A new concept of "genre class" allows us to regard
the two sorts of project as interchangeable, and similarly the two sorts of
extension: see the function `Genres::equivalent`. This is important because
it means that when the build manager searches for an extension, setting the
genre in the `inbuild_requirement` to `extension_genre`, it will return matches
of either `extension_genre` or `extension_bundle_genre`. (This issue had not
arisen for projects, which similarly have two different genres, because inbuild
never needs to search for those as dependent parts of other projects. Well:
if it ever does, it will now get that right.)

Two further pieces of metadata are now meaningful. First, the extension can
activate or deactivate language features. For example, this activates the
feature `performance styles`:

	{
		"is": {
			"type": "extension",
			"title": "Bananas",
			"author": "Graham Nelson",
			"version": "2"
		},
		"activates": [ "performance styles" ]
	}

Note that if the language feature is one which drastically affects the way
source text is broken up into sentences, this will not work as hoped, because
the source text has already had to be broken up in order to find out which
extensions are needed: so it's too late.

More usefully, the extension can include its own private kits.

	{
		"is": {
			"type": "extension",
			"title": "Bananas",
			"author": "Graham Nelson",
			"version": "2"
		},
		"needs": [ {
			"need": {
				"type": "kit",
				"title": "SoftFruitKit"
			}
		} ]
	}

Here, the kit `SoftFruitKit` is automatically merged into the build whenever
the extension is included. Note that the search path used by Inform to find the
definition of `SoftFruitKit` is limited to the extension directory itself: in
other words, this feature can only be used to include private kits belonging
to the extension. For example:

	Graham Nelson
		Bananas
			extension-metadata.json
			Inter
				SoftFruitKit
					...
			Source
				Bananas.i7x

`SoftFruitKit` is then a kit like another, except that it cannot include further
extensions, or (again) activate language features which affect how source text
is broken into sentences.

An extension can have multiple kits built in, or of course can have none. Note
that merely placing a kit into the `Inter` subdirectory is not enough: the
instruction that it is needed must appear in the JSON metadata as well.

Inform automatically rebuilds private kits for extensions as needed if their
compiled Inter binaries are out of date (or missing), as compared with their
Inform 6-syntax source code. So in practice you can simply edit the source
for an extension's private kit, click Go in the Inform app, and the kit will
silently (assuming no errors occur) be rebuilt before the project goes on to
be compiled.

### First evolution proposals implemented (9 October 2022)

Quite a few evolution proposals have now been stacked up, but the first few
are beginning to be implemented. As this is done, so the proposals are also
being slightly rewritten, because there's nothing like implementing a specification
for finding out what's wrong with it.

None of these are part of a release yet, so the only way to experiment with
them is by pulling and building the `master` branch to make your own command-line
tools.

* [IE-0006 New annotations for I6 syntax](https://github.com/ganelson/inform-evolution/blob/main/proposals/0006-i6-syntax-annotations.md)
has an unusual status, since it's really just a syntax for how to augment the I6
language should any augmentations be wanted, rather than a proposal actually to
do so. However, the syntax has been made more explicit now, and the `building`
module of `inter` now implements reading these annotations.

As a tentative test, the `+replacing` annotation for I6 functions has now been
implemented. So for example, placing this function in (say) `ExampleKit`...

	+replacing [ SquareRoot num;
		"Nobody cares about square roots, son.";
	];

...causes it to replace the more arithmetical definition of a function with
the same name in `BasicInformKit`. More specifically:

	+replacing(from BasicInformKit) [ SquareRoot num;
		"Nobody cares about square roots, son.";
	];

replaces the definition if it occurs in `BasicInformKit` but not from anywhere
else. This all gives kits an ability to make replacements which they didn't
previously have. The same annotation can also be used in the source text for
a project:

	Include (-
		+replacing [ SquareRoot num;
			"Nobody cares about square roots, son.";
		];
	-).	

This is, in a sense, not a new ability since it was already possible to write:

	Include (-
		[ SquareRoot num;
			"Nobody cares about square roots, son.";
		];
	-) replacing "SquareRoot".

But the new notation allows individual definitions to be marked as replacements
and makes it much easier to make long runs of replacements.

The effect of all of these alternatives is that given something like:

	When play begins:
		showme the square root of 100.

the story should compile and say:

	"square root of 100" = number: Nobody cares about square roots, son.
	1

rather than, as it otherwise would,

	"square root of 100" = number: 10

This works for definitions other than functions, too. If your new kit doesn't
want `INPUT_BUFFER_LEN` to have the conventional value 260, you could define:

	+replacing(from BasicInformKit) Constant INPUT_BUFFER_LEN = 400;

* [IE-0009 Dialogue Sections](https://github.com/ganelson/inform-evolution/blob/main/proposals/0009-dialogue-sections.md)
is almost completely implemented, but not yet documented (except that the proposal
is now very detailed, and will likely become a new chapter of Writing with Inform).

* [IE-0010 Concepts](https://github.com/ganelson/inform-evolution/blob/main/proposals/0010-concepts.md)
is implemented too. IE-0009 requires IE-0010, but not vice versa.

### Refactoring of "plugins" to "compiler features" (1 September 2022)

Inform has for many years had internally named "plugins" providing optional
compiler functionality, though only inside `inform7`, not `inter` or `inbuild`,
and this meant some awkward conditional compilation in the `supervisor` module.
This is now all refactored so that the named compiler features are managed
by the shared services module `arch`: they are thus present across all three
tools. The term "plugin" is now unhelpful, so the typedef name |plugin|
has become |compiler_feature|.

Kits already had the ability to force compiler features to be active or inactive
by naming them in their metadata; projects have now been given the same ability.
See the `inbuild` manual.

The point of all of this is to pave the way for new features to be added in
a tentative way, switched off by default, but available for projects to test.
If a project includes this line in its JSON metadata file (which is optional,
but we're imagining an expert user here):

    "activates": [ "fruit cultivation" ],

...then the compiler feature named `fruit cultivation` will be active during
the compilation of the project.

Of course, there is no such feature name at present, so this would in fact
throw a problem message. But the idea is that the new features specified by
proposals in the Inform evolution repository can be introduced, in some cases
at least, as named features which are inactive by default until they seem to
be working, and then activated by default when ready for public use.

### Kit incremental rebuilding change (28 August 2022)

A snag emerged when building the new release 10.1.1 for Linux. Inbuild, the
build manager inside of Inform, automatically rebuilds kits from source if they
need it, and uses timestamps of files to determine this. If timestamps are
equal, Inbuild errs on the side of rebuilding, because timestamps are fairly
low in fidelity: either file might actually be newer.

Unfortunately, when Linux apps are sandboxed (i.e., are running in a sort of
protective enclosure for security reasons), the inside of the app installation
is reported to have timestamps which are all equal to each other. No doubt
there are good reasons for this, but it means Inbuild tried to rebuild each
kit inside of the app (so, `BasicInformKit` and so on). And this failed, because
the sandboxed directories are read-only. Inform therefore halted because the build
was thwarted.

The new rule is that a kit which is discovered from an internal nest is never
incrementally rebuilt, whatever timestamps might exist. There doesn't seem any
reason to restrict this rule to Linux: the internal nest inside the apps should
be read-only on all of them (and indeed the MacOS sandbox also enforces this).

### Release notes (8 August 2022)

Not a thrilling development, but we continue to get things organised ready for
formal releases of Inform from this repository. As part of that, the entire
[archive of release notes of past Inform releases](version_history.md) is now
converted to Markdown and migrated here.

In the past, changes to Inform were logged in a sort of ebook, and indeed this
could even be downloaded in ePub format. That was before we had this repository,
though, and the ebook is being discontinued. Release notes here are clearly
easier to maintain, and to link to commits and issues.

### State of Inform talk (30 July 2022)

[The text and slides](https://ganelson.github.io/inform-website/talks/2022/07/31/narrascope-iii.html)
from a talk about Inform at the Narrascope III conference give an outline of
expected future developments in the language.

### Memory saving under the Z-machine (17 July 2022)

When compiling to the Z-machine, Inform now adopts the I6 configuration option
$ZCODE_LESS_DICT_DATA=1, and consequently saves 1 byte per dictionary word.

### String escape notations in I6 syntax (30 June 2022)

I6 inclusions in source text, and kit sources, are run through an I6-to-Inter
compiler within the building module of inter, not through the regular I6 compiler.
Up to this point in the beta, the I6-to-Inter compiler recognised only a few
of the very basic string escapes, which led to Jira bug report I7-2156. This
particularly affects non-English IF writers, and especially translators making
language support kits.

This should all now work: see the new inform7 test case I6StringEscapes-G (the
2000th inform7 test case!) and the building-test module test case schemas for
a thorough exercise of these escapes.

For example, the following function included in I7 source text:

	Include (-
		[ Hyperdiacritical;
			print (char) '^', " might be a caret, who knows.^";
			print (address) 'x^', " might be an x', who knows.^";
			print (address) '^//', " might be a ', who knows.^";
			print (char) '@ss', " might be an @ss, who knows.^";
			print (address) 'x@ss', " might be an x@ss, who knows.^";
			print (char) '@{0041}', " might be an A, who knows.^";
			print (address) 'x@{0041}', " might be an xA, who knows.^";
			print "Les @oeuvres d'@AEsop en fran@,cais, mon @'el@`eve!^";
			print "Na@:ive readers of the New Yorker re@:elected Mr Clinton.^";
			print "Gau@ss first proved the Fundamental Theorem of Algebra.^";
			print "@'a@'e@'i@'o@'u@'y@'A@'E@'I@'O@'U@'Y@`a@`e@`i@`o@`u@`A@`E@`I@`O@`U@^a@^e@^i@^o@^u@^A@^E@^I@^O@^U@:a@:e@:i@:o@:u@:y@:A@:E@:I@:O@:U@:Y^";
			print "@~a@~n@~o@~A@~N@~O@,c@,C@\o@\O@ae@AE@et@Et@th@Th@LL@!!@??@<<@>>@ss@oa@oA@oe@OE^";
			print "So @{a9} is a copyright sign, and @{424} is a capital Cyrillic ef, and @{25B2} is a triangle^";
			print "Backslash: @@92 At sign: @@64 Caret: @@94 Tilde: @@126^";
		];
	-).

prints, if executed (on Glulx - the Z-machine does not support four of these Unicode characters):

	^ might be a caret, who knows.
	x' might be an x', who knows.
	' might be a ', who knows.
	ß might be an ß, who knows.
	xß might be an xß, who knows.
	A might be an A, who knows.
	xa might be an xA, who knows.
	Les œuvres d'Æsop en français, mon élève!
	Naïve readers of the New Yorker reëlected Mr Clinton.
	Gauß first proved the Fundamental Theorem of Algebra.
	áéíóúýÁÉÍÓÚÝàèìòùÀÈÌÒÙâêîôûÂÊÎÔÛäëïöüÿÄËÏÖÜŸ
	ãñõÃÑÕçÇøØæÆðÐþΣ¡¿«»ßåÅœŒ
	So © is a copyright sign, and Ф is a capital Cyrillic ef, and ▲ is a triangle
	Backslash: \ At sign: @ Caret: ^ Tilde: ~

The syntax recognised for character, dictionary and string literals now matches
the syntax recognised by the main I6 compiler, except for one extension: the
I6-to-Inter compiler also allows "[unicode N]", where N is a decimal number,
to mean the character whose code point is N. The reason for this extension to
the syntax is that it means that:

	Include (-
		[ Diacritical;
			print "Ф is a capital Cyrillic ef, and ▲ is a triangle.^";
		];
	-).

will work the same way when run through I7 as the definition:

	[ Diacritical;
		print "Ф is a capital Cyrillic ef, and ▲ is a triangle.^";
	];

would work if found in the source code for a kit -- in both cases, the ef
and the triangle will be passed successfully through. Consistency between
inform7 and inter seems more important on this than consistency between inter
and inform6.

### Withdrawal of -kit, but not of -basic (27 June 2022)

Up to this point, the beta of inbuild (and hence also of inform7) had a
command-line switch "-kit". This told Inbuild that the named kit should be
included in any build of a project to be specified later on the command line.
For example,

	$ inbuild -kit BasicInformKit -kit MyMagicKit -build -project MyProject.inform

The convention was that if no "-kit" was supplied, then the project would
either include just BasicInformKit (if -basic was also given as an option)
or else BasicInformKit and CommandParserKit. (These would then cause
other kits to be included, such as WorldModelKit, and there would also be
a kit to support the language of play, such as EnglishLanguageKit.)

This worked, but was clumsy. Users of the Inform apps could only take advantage
by writing these command-line settings into both an "inform7-settings.txt" and
"inbuild-settings.txt" file, and even then this was finicky (see Jira bug I7-2161).
The command line is anyway not a good place to specify metadata which properly
belongs to a project, and also had no way to express version numbers for the
kits desired.

"-kit" has now been abolished. "-basic" remains, and works as before.

So how do you specify that a project expects to see a kit? The answer is to
place a suitable project_metadata.json file into the project's materials directory.
See [A Guide to Project Metadata](https://ganelson.github.io/inform/inbuild/M-agtpm.html).

### Incremental building of kits enabled (26 June 2022)

This is arguably a bug fix, but is not directly a fix of any currently open bug
in Jira (but see closed I7-2155). It was always intended that inbuild and
inform7 would incrementally build any kits they need in order to build Inform
projects: that has worked up to now with the standard kits, but in many
circumstances not with newly written kits, such as those intended to be used
with language extensions. (It's complicated: some ways of using inbuild did
in fact do this, but inform7 did not.)

This is now enabled. On the console (or in the app's Console pane), you will
now see lines like the following when a kit is being rebuilt by inform7 itself:

	(Building FrenchLanguageKit for architecture 16)
	(Building FrenchLanguageKit for architecture 16d)
	(Building FrenchLanguageKit for architecture 32)
	(Building FrenchLanguageKit for architecture 32d)

A side-effect of all this is that the output of inbuild -graph for a project
is now substantially longer. This is basically because a project does indeed
depend on lots of source files, and the new larger graph is a truer picture
than the old ones. But we may eventually work out more concise ways to print it.

### Language metadata respecified as JSON (23 June 2022)

As has been noticed already (see e.g. Jira bug I7-2155), the new Inbuild has
been fairly sketchy in its handling of language bundles, and of using Inform
to make non-English IF. In particular, language kits such as FrenchLanguageKit
(if provided) were not being loaded automatically and instead required -kit
to be used at the command line, or with a settings file; and even then,
EnglishLanguageKit was being loaded as well, causing definition clashes.

This should all now be resolved.

Language bundles have now been redesigned, and are documented in the Inbuild
manual, which has been rewritten today: see [A Guide to Language Bundles](https://ganelson.github.io/inform/inbuild/M-agtlb.html).
Like kits, language bundles are adopting the new JSON metadata format.

### Change of testing UUID (22 June 2022)

A change to the batch-testing tool Intest means that the UUID used for Inform
projects being tested is now 00000000-0000-0000-0000-000000000000: this is meant
to be a visibly bogus value, replacing 0B00B00D-3307-4688-B2D8-95DB962781B4.
UUIDs are in theory unique per project, but Intest operates Inform projects with
this single bogus UUID for every test case it runs.

As a result, all six inblorb test cases and 9 of the inform7 tests will fail
(1=Awkward 2=Audiovisual 3=Fancy-Z 4=Plain-Z 5=Plain 6=Ingredients 7=Fancy
8=Index-Card 9=Index-Card2) by showing a discrepancy on the UUID somewhere
unless you have pulled the latest Intest.

### Kit metadata respecified as JSON (4 June 2022)

Kits are new in 10.1, and are documented in the Inbuild manual, which has been
rewritten today: see [A Guide to Kits](https://ganelson.github.io/inform/inbuild/M-agtk.html).

The change today is that the "kit_metadata.txt" file has now changed format and
name, to become "kit_metadata.json". This JSON format is a work in progress and
is likely to become a uniform way to describe resources used by Inbuild (extensions,
language bundles, interpreters, website templates, and so on): the point of having
such a format is to prepare the way for Inbuild to understand remote resources
somewhere on the Internet. For now JSON metadata files exist only for kits, but
some of the plumbing has been put in for general resources to have them.

This required new APIs in the foundation library at the Inweb repository to
read, write and validate JSON files: see the new section [JSON](https://ganelson.github.io/inweb/foundation-module/4-jsn.html).
You'll therefore need to pull the latest Inweb in order to build the latest Inform.

The current set of requirements for what Inbuild will read as resource metadata is
at [inform7/Internal/Miscellany/metadata.jsonr](inform7/Internal/Miscellany/metadata.jsonr).
There are also a few semantic constraints: see [JSON Metadata](https://ganelson.github.io/inform/supervisor-module/2-jm.html).

### Versioning policy for the built-in kits (4 June 2022)

There are currently five kits supplied in the Inform installation: 

	WorldModelKit
	EnglishLanguageKit
	CommandParserKit
	BasicInformKit
	BasicInformExtrasKit

Up to now these had no version numbers, but we want to encourage all kits to
have semantic version numbers, so we should comply with that ourselves.

In practice these kits are tightly coupled to the main compiler: (a) many bug fixes
in Inform which users think of as compiler fixes are in fact changes to these kits;
and (b) in normal circumstances people will never want to use the kits from one
build with the compiler from another. So the most useful way to describe the version
of a built-in kit is to identify which compiler release it came with.

As a policy decision, the version numbers of these five kits are therefore going
to be the same as those of the main compiler. (Today, that's "10.1.0-beta+6V21".)

This is enforced by changes to the makescript used when updating the main compiler's
version number, which uses a new feature of the Inpolicy tool:

	inpolicy/Tangled/inpolicy -sync-kit-versions

For its implementation, see: [Kit Versioning](https://ganelson.github.io/inform/inpolicy/2-kv.html).

User-written kits not included in the Inform installation should not do this.
They won't be tied to particular compiler releases, and should have their own
semantic version numbers reflecting their own history of development.

### Incorporation of "6M62 Patches" (31 May 2022)

In recent years Inform users have been keeping an extension called "6M62 Patches",
which contained a small selection of bug fixes to the template files of Inform 6
code in build 6M62 (i.e., Inform version 9.3). Those template files have now
become kits, the Inform 6 code has become Inter code, and changes to the way the
"Include..." sentence works mean that "6M62 Patches" is not compatible with v10.1.

It would be easy enough to fix that, but there is now no need: all bug fixes
in "6M62 Patches" have been adopted in the kits for v10.1. Any old project which
included "6M62 Patches" can now stop doing so.

It was a real public service to create and maintain "6M62 Patches", and we thank
everyone who contributed to it: their work lives on.