One-time file loading proposal

One-time file loading

Poll standings

See below for voting instructions.

Systems

[ ] conforms to ANS Forth.
  Gforth
  VFX Forth
  Forth6 cross compilers (Stephen Pelc)
  PFE (Guido Draheim)
[ ] already implements the proposal in full since release [ ]:
  Gforth, all releases
[ ] implements the proposal in full in a development version:
[ ] will implement the proposal in full in release [ ].
  VFX Forth 4
  PFE 33.67
[ ] will implement the proposal in full in some future release.
  Forth6 cross compilers
There are no plans to implement the proposal in full in [ ].
  4th V3.5a (Hans Bezemer)
  MacForth (Ward McFarland)
[ ] will never implement the proposal in full:

Programmers

[ ] I have used (parts of) this proposal in my programs:
  Hans Bezemer
  Anton Ertl
  Robert Epprecht
  David N. Williams
[ ] I would use (parts of) this proposal in my programs if the systems
    I am interested in implemented it:
  Stephen Pelc
[ ] I would use (parts of) this proposal in my programs if this
    proposal was in the Forth standard:
  Stephen Pelc
  Charles Mélice
[ ] I would not use (parts of) this proposal in my programs.

Informal results

PFE already implemented most of the proposal, but with the name REQUIRES instead of REQUIRE.

Change History

2006-02-12

Added discussion on whether REQUIRED should reinclude files that were INCLUDED.

2006-01-21

Normative: Added memory allocation restriction to INCLUDED (and a related section in the informative parts).
Normative: After forgetting the contents of a file with FORGET or a marker, REQUIRED will include the file again (plus an informative section on that; the reference implementation does not implement that yet).
Normative: Put the new words in the FILE EXT word set.
Normative: replaced "perform word" with "perform the function of word".
Recorded answers to the question about the name of REQUIRE/NEEDS/REQUIRES, and corrected information about REQUIRES in MPE systems (it does not perform what is proposed).
Added a section on INCLUDE interacting with BASE.
Added information about features to increase the efficiency of the C approach.

2001-08-14

Changed stack effect from ( i*x c-addr u -- j*x ) to ( i*x c-addr u -- i*x ), following a suggestion from Guido Draheim, and related changes in the text.

Problem

A library is needed by several parts of the source code (e.g., in other libraries), but it should be loaded only once.

Proposal

Add the following restriction to the definition of INCLUDED:

INCLUDED may allocate memory in data space before it starts interpreting the file.

In the definition of INCLUDED, change Text interpretation begins at the file position where the next file read would occur into Text interpretation begins at the start of the file.

Define the following words:

REQUIRED   FILE-EXT ( i*x c-addr u -- i*x )

If the file specified by c-addr u has been INCLUDED or REQUIRED already, but not between the definition and execution of a marker (or equivalent usage of FORGET), discard c-addr u; otherwise, perform the function of INCLUDED.

An ambiguous condition exists if a file is REQUIRED while it is being REQUIRED or INCLUDED.

An ambiguous condition exists, if a marker is defined outside and executed inside a file or vice versa, and the file is REQUIRED again.

An ambiguous condition exists if the same file is REQUIRED twice using different names (e.g., through symbolic links), or different files with the same name are REQUIRED (by doing some renaming between the invocations of REQUIRED).

An ambiguous condition exists if the stack effect of including the file is not ( i*x -- i*x ).

REQUIRE   FILE-EXT ( i*x "name" -- i*x )

Skip leading white space and parse name delimited by a white space character. Push the address and length of the name on the stack and perform the function of REQUIRED.

INCLUDE   FILE-EXT ( i*x "name" -- j*x )

Skip leading white space and parse name delimited by a white space character. Push the address and length of the name on the stack and perform the function of INCLUDED.

Typical Use

... s" filename" required ...
require filename
include filename

Remarks

Changes to `INCLUDED`

The added restriction codifies an existing environmental restriction in widely-used systems (at least Win32Forth and MPE's systems) that has not led to problems in programs. In the context of REQUIRED this restriction allows a simpler implementation. In effect this restriction means that colon definitions or contiguous data regions cannot extend across calls to INCLUDED (one can use INCLUDE-FILE instead).

As an alternative to adding this restriction, we could drop the requirement that REQUIRED does not re-include files that were INCLUDED. People probably do not use REQUIRED and INCLUDED on the same file anyway. The disadvantages of this alternative are: the result is probably confusing in cases where people do use REQUIRED and INCLUDED on the same file; and AFAIK all systems that implement REQUIRED (or similar words) support that requirement.

Should we also add a restriction that allows INCLUDED to allocate data space after the interpretation of the included file ends?

The change in the definition fixes an obvious copy-and-paste error.

REQUIRED syntax

The syntax follows the good example of INCLUDED in being non-parsing. That syntax allows more flexible uses and it allows REQUIREing filenames containing spaces.

INCLUDE

INCLUDE is implemented and used widely, so we might just as well standardize it (and why not in this RfD). OTOH, once REQUIRE is widely implemented, it might see much less use. if enough people argue against its inclusion, I will remove it from this RfD.

INCLUDE and BASE

There are supposedly systems that set BASE to decimal at the start of INCLUDE. One could accomodate such systems by adding a restriction:

An ambiguous condition exists if the value of BASE is not (decimal) ten.

However, IMO it is confusing if INCLUDE silently changes BASE and INCLUDED does not, so I will only add this restriction if somebody can name such a system (that is a necessary, not a sufficient condition).

A better approach IMO is to warn the user if BASE is not decimal when starting INCLUDE, and if the BASE has changed when ending INCLUDE (and probably for INCLUDED and INCLUDE-FILE, too).

REQUIRE

Given what happened with INCLUDED and INCLUDE, we might just as well provide a parsing variant of REQUIRED right from the start. However, the essential word is REQUIRED, the parsing variant is just syntactic sugar.

REQUIRE name

The name is modeled on the relation between INCLUDED and INCLUDE, and also on Emacs Lisp's require. However, this word has been implemented under other names: NEEDS in various systems, REQUIRES in PFE.

What's worse, some systems have used these names for other purposes: MPE's systems have REQUIRES with a different meaning, PFE has a NEEDS with a different meaning, and ciforth has a REQUIRE with a different meaning (but ciforth's author thinks that ciforth is not significant and has promised to rename/drop his word if consensus on using REQUIRE for another purpose is reached <ielhfd.416@spenarnc.xs4all.nl>).

So, which of the names would you prefer? Please post or mail me your preferences, and I will collect them and then decide on the final name, if any. Also, if you know of other conflicts for these names, please let me know.

Results from the RfDs:

Stephen Pelc (MPE) can live with REQUIRED and REQUIRE. REQUIRES would conflict with MPE's systems.
Alex McDonald (Win32Forth) can live with REQUIRE <1137148722.722401.233900@f14g2000cwb.googlegroups.com>; currently Win32Forth supports NEEDS and REQUIRES (with ideas to drop REQUIRES).

Why not use load screens?

Some people prefer to have a single file that contains INCLUDEs for all the other files in a program. These people do not need REQUIRED.

However, other people want to build programs out of reusable (and possibly independently developed) libraries, and the load-screen approach causes increased maintenance work in this context: E.g., if a new version of a library needs to load an additional sublibrary, the load screens of all programs using the library would have to be changed. In contrast, with REQUIRED, the library REQUIREs the sublibrary itself, and REQUIRED makes sure that it is not loaded twice.

What about the C approach

Some might say: Why not use the C solution: The C solution to this problem is putting a wrapper like

#ifndef FILE_H
#define FILE_H
...
#endif

around every source file. This is inefficient (the whole file has to be read again, unless the compiler does some pretty sophisticated stuff), and requires cooperation from the author of the file (which is problematic, because not the author, but the users of the file have the trouble).

SwiftForth has a word \\ that can be used to mostly eliminate the inefficiency mentioned above: it ends interpreting the current file; Win32Forth has an equivalent word \s. SwiftForth also has a word OPTIONAL that supports the C-like technique <43bc4e10$0$14122$ba620e4c@news.skynet.be>, <43c12a85$0$29455$ba620e4c@news.skynet.be>.

Using the dictionary for implementation

One thing that could be done to make the implementations simpler is to specify that INCLUDED and REQUIRED can change the dictionary before or after text-interpreting the file; that would allow the implementation to store the names of the included files in a wordlist. The downside would be that INCLUDED could not be used for including parts of colon definitions or the data part of CREATEd words. However, such uses are probably rare (if they occur at all in existing code), and they can still be performed (although in a more cumbersome way) with INCLUDE-FILE. What do you think?

What about MARKER and FORGET?

The proposed behaviour is probably what the users are expecting, and AFAIK all implementations of such words already support that behaviour. The ambiguous condition allows different implementation approaches, and the usage that is explicitly unspecified in this way probably does not matter to users.

Experience

Many systems have REQUIRED, REQUIRE, REQUIRES, and/or NEEDS. Many systems contain INCLUDE. The occurences of these words in the Gforth sources are:

occurences regexp
 88        "^require "
 45        "^include "
  4        " included$"
  2        " required$"

Implementation and Tests

Reference implementation (does not deal as proposed with markers and FORGET).
Tests

Comments

Albert van der Horst suggested that REQUIRE should in some way be able to use LOAD <isj83k.igv@spenarnc.xs4all.nl>. However, I believe that such a word, if needed, should have a different name.

Hans Bezemer writes that 4th has a parsing [NEEDS (apparently used like "[NEEDS file name]"). 4th also has INCLUDE <1136473379.719600.193630@g49g2000cwa.googlegroups.com>.

Stephen Pelc:

In order to ease use of library code, it may be as well to 
consider specifying how the following items are handled when the 
file/source is INCLUDED:
  BASE   decimal?
  search order
  ...

Phil Budne writes that SNOBOL4 has a -INCLUDE that behaves like the proposed REQUIRE.

from an earlier (Pre-RfD) version of the proposal

Michael L. Gassanenko:

Yes. I do use NEEDS.

Peter Knaggs:

Perl
introduced a version of REQUIRED some time ago that works exactly in
this manner.  I agree very much with its usefulness, indeed it would
allow a standard "library" model.  Having said that, it can be defined
using standard ANS.

Guido Draheim:

* Both Forth.com's Swiftforth and MPE's ProforthVFX have defined
`requires` as the selfparsing version of `required`, and they use it
[It turns out that REQUIRES in MPE's systems has a different
meaning]. PFE will adopt this in the next version (> 0.30.30)

* The specification is non-deterministic in its stack effect - the user of
`required` has no way to check in advance if a file will get `included`.
Many systems use a word like `loaded?` but it can not as easily specified
for many systems just as it is done for `required` itself, where the form of
`requires` has found wide acceptance. It may be useful to *allow* the use 
CSP-like techniques to ensure a deterministic stack-effect, and it may be
useful to *recommend* that atleast a warning message is shown for the case
of stack-depth differences.

Voting instructions

Fill out the appropriate ballot(s) below and mail it/them to me <anton@mips.complang.tuwien.ac.at>. Your vote will be published (including your name (without email address) and/or the name of your system) here. You can vote (or change your vote) at any time by mailing to me, and the results will be published here.

Note that you can be both a system implementor and a programmer, so you can submit both kinds of ballots.

Ballot for systems

If you maintain several systems, please mention the systems separately in the ballot. Insert the system name or version between the brackets. Multiple hits for the same system are possible (if they do not conflict).

[ ] conforms to ANS Forth.
[ ] already implements the proposal in full since release [ ].
[ ] implements the proposal in full in a development version.
[ ] will implement the proposal in full in release [ ].
[ ] will implement the proposal in full in some future release.
There are no plans to implement the proposal in full in [ ].
[ ] will never implement the proposal in full.

If you want to provide information on partial implementation, please do so informally, and I will aggregate this information in some way.

Ballot for programmers

Just mark the statements that are correct for you (e.g., by putting an "x" between the brackets). If some statements are true for some of your programs, but not others, please mark the statements for the dominating class of programs you write.

[ ] I have used (parts of) this proposal in my programs.
[ ] I would use (parts of) this proposal in my programs if the systems
    I am interested in implemented it.
[ ] I would use (parts of) this proposal in my programs if this
    proposal was in the Forth standard.
[ ] I would not use (parts of) this proposal in my programs.

If you feel that there is closely related functionality missing from the proposal (especially if you have used that in your programs), make an informal comment, and I will collect these, too. Note that the best time to voice such issues is the RfD stage.

Anton Ertl