Methodology

The atomic unit of CE is the paragraph. Broad references to a previous paragraph are not allowed; sentences may refer to previous sentences within the same paragraph.

From that unit we can treat rule-and-lexicon work as approximation of an ideal set. The current version is a computable stand-in. The job is to close the gap.

Ideal CE

Imagine a space P of all possible paragraphs: every finite string of UTF-8 characters. Inside it sits a subset.

Define CEi ⊂ P, “ideal CE”, as the set of all paragraphs that are unambiguous with respect to sense and assignment: well-formed English that admits exactly one reading under the rule set, is grammatically correct, and uses only words in the ideal lexicon. If p ∈ CEi, the paragraph is correct; otherwise it is incorrect.

CEi is not directly observable, because the rules are not written down in advance. It is a Stewart set: a paragraph is in or out by “I know it if I see it” (Justice Potter Stewart, 1964). Different readers may disagree on whether a paragraph is correct, in some edge cases; these paragraphs are said to be in the "zone of subjectivity".

Computable Approximations

To estimate CEi, we start with CE1.0.0: a small vocabulary v and a simple rule set r. From those we can decide, for every paragraph p, whether p is in CE1.0.0. That decision is computable; the set is no longer a Stewart set.

CE1.0.0 ≠ CEi. The work is to develop successive iterations CE1.0.x, for x ∈ {0, 1, 2, 3, …}, each with an improved rule set and vocabulary, so that CE1.0.x approaches CEi.

CE1.0.x overlaps CEi but also occupies space outside it. That exterior is over-acceptance: paragraphs the current rules pass that ideal CE would reject.

Venn diagram of all possible paragraphs P. The left circle is ideal CE (CEi); the right circle is the current implementation CE1.0.0. Their overlap is correct acceptance, illustrated by the paragraph “Dogs eat food.” The CEi-only region is false negatives (over-rejection). The CE1.0.0-only region is false positives (over-acceptance). Outside both circles, “The and dog.” is correctly rejected.
P is every possible paragraph. CEi is the unobservable ideal. CE1.0.0 is the current computable approximation. The mismatch is false positives and false negatives.

p ∈ CE1.0.x ⇔ f(p) = 1 for all f ∈ CE1.0.x’s rule set, where each rule returns pass (1) or fail (0).

FP = CE1.0.x − (CEi ∩ CE1.0.x) = false positives, over-acceptance (pass but incorrect)

FN = CEi − (CEi ∩ CE1.0.x) = false negatives, over-rejection (fail but correct)

Algorithm for Improvement

  1. Define a set T to hold test paragraphs. Start empty: T = ∅.
  2. Develop a test paragraph p. Test paragraphs can be sampled from large corpora. It is easy to generate p ∈ FP because most of P is incorrect. It is also easy to generate p ∈ FN by choosing a paragraph that uses a word in CEi’s lexicon that is not yet in CE1.0.x.
  3. Check p against the current rule set.
    • If p fails only on the lexicon, add the word to v.
    • If p passes but is incorrect (p ∈ FP), add a rule to CE1.0.x that excludes it.
    • If p fails but is correct (p ∈ FN), weaken or delete an existing rule in CE1.0.x.
    Then check every paragraph already in T against the new rule set. The change must not regress the existing test set.
  4. Add p to T and return to step 2.

Measuring Accuracy

Closeness of CE1.0.x to CEi is measured empirically as the rates of false negatives and false positives on a held-out set T₂ of wild-English paragraphs that are not yet in T.

If T₂ has 100 correct paragraphs and 100 incorrect paragraphs, we might see 13% FN and 10% FP, which is (87% + 90%) / 2 = 88.5% accurate.

The goal is a CE1.0.x that is as close as possible to 100% on any test set T. That is equivalent to defining CEi objectively rather than subjectively.

Armed with such a rule set and lexicon, noah can deterministically decide whether any paragraph p is CE.

Grammaticality judgments near the boundary will eventually vary speaker by speaker; that is a well-established fact of natural language. Past a certain percentage, subjectivity makes it impossible to measure whether a paragraph is correct. In that zone around the edge of CEi, the boundary is fuzzy, or probabilistic, with disagreement even among expert native speakers. CE still makes a sharp, explicit determination there: a necessary over-specification. That counts as success in defining CE.

Contributing to the Next Version of CE

If you find an example of a false positive or false negative, please submit a pull request that includes:

We will try to incorporate every pull request judged sound into the next version of CE.

CE also forces a proposed rule to execute inside noah-linter and to pass the existing regression suite of rules that already exist. That formality is how later versions asymptotically approach as perfect a set of rules for English as possible. The overview states the linguist-facing case.

Back to Canonical English