Getting Started with Overlay Language

The Step 4 Python code works, but it has a structural problem: business logic and I/O are tangled together. Consider user_id in HttpHandlers:

def user_id(request: BaseHTTPRequestHandler) -> int:
    return int(request.path.split("/")[-1])

This one function mixes three concerns: reading request.path (I/O), splitting on "/" (a business decision about URL format), and parsing the last segment as an integer (another business decision). The path separator, SQL queries, and format templates are all hardcoded in Python — changing any of them means changing Python code.

The Overlay language solves this by separating the application into three layers:

• Python FFI wraps individual stdlib calls in @scope adapters — one class per operation (sqlite3.connect, str.split, wfile.write). Each adapter declares its inputs as @extern and exposes a single @public @resource output. The adapter contains zero business logic.

• ``.oyaml`` files contain all application logic. Overlay is not just a configuration format — it is a complete language with lexical scoping, nested scopes, deep-merge composition, and lazy evaluation. These features make it more natural than Python for expressing business logic, which is inherently declarative (“the user ID is the last URL segment”, “the response format is total={total} current={current}”).

• Configuration values (SQL queries, format strings, host/port) are pure YAML scalars, gathered in one place.

Business logic written in .oyaml is portable: it is decoupled from the Python FFI layer. Swap the FFI adapters and the same .oyaml logic runs against a different runtime — mock adapters for unit testing, a different language’s stdlib for cross-platform deployment, or instrumented adapters for profiling. With Python @scope decorators, business logic is locked to the Python runtime and cannot be extracted or retargeted.

Below is the same Step 4 web application rewritten in this style.

Python FFI adapters

Each @scope class wraps exactly one stdlib operation. Below are three representative adapters; the full module (tests/fixtures/app_oyaml/stdlib_ffi/FFI.py) contains 10 more following the same pattern.

SqliteConnectAndExecuteScript — multiple inputs, single output:

"""sqlite3.connect(database_path) + executescript(setup_sql) -> connection"""

import sqlite3

from overlay.language import extern, public, resource


@extern
def database_path() -> str: ...


@extern
def setup_sql() -> str: ...


@public
@resource
def connection(database_path: str, setup_sql: str) -> sqlite3.Connection:
    conn = sqlite3.connect(database_path, check_same_thread=False)
    conn.executescript(setup_sql)
    return conn

GetItem — generic sequence[index] operation:

"""sequence[index] -> element"""

from overlay.language import extern, public, resource


@extern
def sequence() -> object: ...


@extern
def index() -> int: ...


@public
@resource
def element(sequence: object, index: int) -> object:
    return sequence[index]  # type: ignore[index]

HttpSendResponse — chained I/O, send status + headers + body:

"""send_response(status_code) + end_headers() + wfile.write(body) -> written"""

from http.server import BaseHTTPRequestHandler

from overlay.language import extern, public, resource


@extern
def request() -> BaseHTTPRequestHandler: ...


@extern
def status_code() -> int: ...


@extern
def body() -> bytes: ...


@public
@resource
def written(
    request: BaseHTTPRequestHandler, status_code: int, body: bytes
) -> BaseHTTPRequestHandler:
    request.send_response(status_code)
    request.end_headers()
    request.wfile.write(body)
    return request

Notice what is not here: no SQL queries, no "/" separator, no format string, no :memory: path, no port number. Those are all business decisions that live in the .oyaml.

.oyaml composition

An .oyaml file describes a dependency graph, not an execution sequence. There is no top-to-bottom control flow — the runtime evaluates resources lazily, on demand. Think spreadsheet cells, not shell scripts.

The business logic lives in Library.oyaml, which references FFI adapters through abstract declarations (FFI: scope with [] slots). A concrete FFI module (stdlib_ffi/FFI.py) overrides these slots at composition time. This separation means the business logic is portable — swap stdlib_ffi for a different FFI implementation and the .oyaml files need no changes.

The following sections walk through Library.oyaml one scope at a time, introducing new language concepts as they appear.

SQLiteDatabase — extern, inheritance, wiring, projection

SQLiteDatabase:
  database_path: []                         # extern: caller must provide this
  setup_sql: []                             # extern: caller must provide this
  _db:
    - [FFI, SqliteConnectAndExecuteScript]  # inherit the FFI adapter
    - database_path: [database_path]        # wire extern → adapter input
      setup_sql: [setup_sql]
  connection: [_db, connection]             # projection: expose _db.connection

Four new concepts:

• ``field: []`` — an extern declaration, the .oyaml equivalent of @extern. The value must come from a parent scope or the caller.

• ``- [FFI, SqliteConnectAndExecuteScript]`` — inheritance. _db inherits the FFI adapter, gaining all of its resources (connection).

• ``database_path: [database_path]`` — wiring. The reference [database_path] is a lexical lookup: search outward through enclosing scopes until a field named database_path is found.

• ``connection: [_db, connection]`` — path navigation. Access the connection resource on the child scope _db. The leading underscore on _db makes it private; connection is the public-facing projection.

UserRepository (app-scoped part) — nested scope, scope-as-dataclass

UserRepository:
  connection: []                            # extern: from SQLiteDatabase
  user_count_sql: []                        # extern: provided by app

  User:                                     # scope-as-dataclass
    user_id: []
    name: []

  _count:
    - [FFI, SqliteScalarQuery]
    - connection: [connection]
      sql: [user_count_sql]
  user_count: [_count, scalar]

• ``User:`` is a nested scope with two extern fields — the .oyaml equivalent of @scope class User with @public @extern fields. It acts as a dataclass constructor: current_user (below) will supply values for user_id and name.

• ``connection: []`` declares that UserRepository expects a connection from outside. When composed with SQLiteDatabase inside app, this extern is satisfied by SQLiteDatabase.connection — resolved by name through lexical scoping.

UserRepository.RequestScope — ANF style, cross-scope references

  RequestScope:
    user_id: []                             # extern: from HttpHandlers
    user_query_sql: []                      # extern: from app.RequestScope

    _params:
      - [FFI, TupleWrap]
      - element: [user_id]

    _row:
      - [FFI, SqliteRowQuery]
      - connection: [connection]            # lexical: finds UserRepository.connection
        sql: [user_query_sql]
        parameters: [_params, wrapped]

    _identifier:
      - [FFI, GetItem]
      - sequence: [_row, row]
        index: 0
    _name:
      - [FFI, GetItem]
      - sequence: [_row, row]
        index: 1

    current_user:
      user_id: [_identifier, element]
      name: [_name, element]

    current_user_name: [current_user, name]

This section demonstrates A-Normal Form (ANF): every intermediate result must be bound to a named field. You cannot write GetItem(sequence=SqliteRowQuery(...).row, index=0) — instead, _row holds the query result, and _identifier extracts column 0 from _row.row. The cost is verbosity; the benefit is that every intermediate value is inspectable and independently composable.

Cross-scope lexical reference: connection: [connection] inside RequestScope finds UserRepository.connection — the lexical scope chain searches outward through parent, grandparent, etc. No import statement is needed; the scope hierarchy is the namespace.

Constructing ``current_user``: Instead of calling User(user_id=..., name=...), the .oyaml directly defines current_user as a scope with two fields. The User scope-as-dataclass above establishes the field names; here those same names are filled with concrete values.

HttpHandlers — flat inheritance, qualified this

HttpHandlers:
  user_count: []                            # extern: from UserRepository

  RequestScope:
    - [FFI, ExtractUserId]                  # provides: user_id
    - [FFI, HttpSendResponse]              # provides: written
    - request: []                           # extern: injected per-request
      path_separator: []                    # extern: from app.RequestScope
      response_template: []                 # extern: from app.RequestScope
      status_code: 200                      # inline scalar
      current_user_name: []                 # extern: from UserRepository.RequestScope

      _format:
        - [FFI, FormatResponse]
        - response_template: [response_template]
          user_count: [user_count]
          current_user_name: [current_user_name]
      response_body: [_format, response_body]
      body: [response_body]

      # Qualified this: fails loudly if HttpSendResponse is not composed,
      # unlike `written: []` which silently creates an empty scope.
      response: [RequestScope, ~, written]

Flat inheritance: RequestScope inherits two FFI adapters in its inheritance list (- [FFI, ExtractUserId], - [FFI, HttpSendResponse]). Their @extern and @resource fields all merge into RequestScope’s own field namespace. The last list item (the mapping starting with request: []) defines RequestScope’s own fields.

Lexical scoping across scope boundaries: [user_count] inside RequestScope searches outward and finds HttpHandlers.user_count. At this point user_count is just an extern [] — its actual value comes from UserRepository after deep merge (explained below).

Qualified this: ``[RequestScope, ~, written]`` — instead of declaring written: [] and writing response: [written], this navigates the runtime composition graph to access the written property inherited from HttpSendResponse. The advantage: if HttpSendResponse is accidentally not composed, this fails with an error instead of silently creating an empty scope.

NetworkServer — deep merge, config scoping

NetworkServer:
  - [FFI, HttpServerCreate]
  - host: []
    port: []
    RequestScope: []
    _handler:
      - [FFI, HttpHandlerClass]
      - RequestScope: [RequestScope]
    handler_class: [_handler, handler_class]

All the scopes above live in Library.oyaml. They reference [FFI, Xxx] which resolves to abstract declarations at the top of the file — no concrete Python code is involved yet.

Apps.oyaml — integration entry point

Apps.oyaml is a separate file that inherits the real FFI implementation and the Library, then defines concrete application entries:

# Apps.oyaml — integration entry points: Library + real FFI + configuration values.
# Named "Apps" (plural) because multiple entry points can coexist here.
- [stdlib_ffi]                          # inherit real Python FFI adapters
- [Library]                             # inherit business logic
- memory_app:
    - [Apps, ~, SQLiteDatabase]         # qualified this: inherited from Library
    - [Apps, ~, UserRepository]
    - [Apps, ~, HttpHandlers]
    - [Apps, ~, NetworkServer]
    - database_path: ":memory:"
      setup_sql: |
        CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT);
        INSERT INTO users VALUES (1, 'alice');
        INSERT INTO users VALUES (2, 'bob');
      user_count_sql: "SELECT COUNT(*) FROM users"
      host: "127.0.0.1"
      port: 0
      RequestScope:
        user_query_sql: "SELECT id, name FROM users WHERE id = ?"
        path_separator: "/"
        response_template: "total={total} current={current}"

Library/FFI separation: - [stdlib_ffi] makes the real FFI module (Python @scope classes) visible. - [Library] makes the business logic visible. When composed, stdlib_ffi.FFI (real implementations) deep-merges with Library.FFI (abstract declarations), and the real @resource methods override the [] slots. The business logic never imports Python directly.

Portability: To run the same business logic on a different runtime, replace - [stdlib_ffi] with a different FFI package — the Library.oyaml file needs no changes. To test with mocks, provide a mock FFI module instead of stdlib_ffi.

“What Color Is Your Function?” (blog post): The same Library.oyaml runs unchanged on both synchronous and asynchronous runtimes. Replacing - [stdlib_ffi] with - [async_ffi] swaps the FFI layer to one built on aiosqlite + starlette (implementation) — the business logic never knows whether it is sync or async. Function colour is confined entirely to the FFI boundary; Library.oyaml itself is colourless.

Composition via inheritance: memory_app inherits four scopes via qualified this ([Apps, ~, SQLiteDatabase] etc.) because these scopes are inherited properties, not own properties of Apps.oyaml. This is not four separate instances — it is a single scope with all four merged together. The last list item supplies concrete values for every [] extern.

Deep merge: Both UserRepository and HttpHandlers define a RequestScope. When composed inside memory_app, these merge by name into a single RequestScope. After merging:

• user_id (from HttpHandlers.RequestScope via ExtractUserId) becomes visible to UserRepository.RequestScope, which uses it to look up current_user

• current_user_name (from UserRepository.RequestScope) becomes visible to HttpHandlers.RequestScope, which uses it in _format

Neither scope imports or references the other — deep merge makes their fields mutual siblings automatically. This is the most powerful feature of the Overlay language: cross-cutting concerns compose without glue code.

Config value scoping: App-lifetime values (database_path, host, port) live directly in memory_app. Request-lifetime values (user_query_sql, path_separator, response_template) live in memory_app.RequestScope — they are only needed during request handling.

Syntax quick reference

Syntax	Meaning
field: []	Extern — value must be provided by a parent scope or caller
field: [other]	Lexical reference — look up other in the lexical scope chain
field: [child, property]	Path navigation — access property on child
field: [Scope, ~, symbol]	Qualified this — access inherited symbol through the runtime graph
field: "literal"	Scalar value — string, number, etc.
_field: ...	Private — not visible to external callers (leading underscore)
Scope: with a mapping	Nested scope — a child scope with its own fields
Scope: with a list	Inheritance — - [Parent] items are inherited scopes; the last item (a mapping) defines own fields

Python vs Overlay language

Aspect	Python @scope	.oyaml
Composition	Manual @extend + RelativeReference	Inheritance list: - [Parent]
Dependency injection	@extern parameter names	field: [] + lexical scope chain
Expression style	Nested function calls	ANF: every intermediate has a name
Cross-cutting concerns	Explicit adapter / glue code	Deep merge: same-named scopes auto-merge
Accessing inherited members	self.xxx / parameter injection	Qualified this: [Scope, ~, symbol]
Business logic location	Mixed with I/O in Python	Separate .oyaml file, portable across FFI
Configuration	Kwargs at call site	Scalar values in memory_app: scope

Evaluation

import tests.fixtures.app_oyaml as app_oyaml
from overlay.language import evaluate

# evaluate() auto-discovers stdlib_ffi/, Library.oyaml, and Apps.oyaml.
root = evaluate(app_oyaml, modules_public=True)

# Access the composed app — Apps is the .oyaml file name.
composed_app = root.Apps.memory_app

composed_app.server               # HTTPServer on 127.0.0.1:<assigned port>
composed_app.connection           # sqlite3.Connection to :memory:
composed_app.user_count           # 2

# Create a fresh request scope (per-request resources):
scope = composed_app.RequestScope(request=fake_request)
scope.current_user.name           # "alice"
scope.response                    # sends HTTP response as side effect

Swapping configuration is just a different entry in Apps.oyaml — the Python FFI adapters and Library.oyaml never change. Swapping the FFI layer is just a different @scope module — the .oyaml business logic never changes.

Runnable tests for this example are in tests/test_readme_package_examples.py, using the fixture package at tests/fixtures/app_oyaml/.

The full language specification is in Overlay Language Specification.

The semantics of the Overlay language are grounded in the Overlay-Calculus, a formal calculus of overlays.