WIP: ref(lexer): attempt to not try on every token #7

Closed

primrose wants to merge 2 commits from reduce-try-usage into dev

pull from: reduce-try-usage

merge into: elland:dev

elland:dev

elland:pictures

elland:nixify

elland:tentative-change-to-lexing-order

elland:runner

elland:main

Conversation 3 Commits 2 Files changed 2 +87 -27

primrose commented 2025-09-28 04:43:29 +00:00

Collaborator

Copy link

What do we do with invalid tokens? I think we should make them fail and not even lex.

The question is whether we should lex >/foo/ as Token and leave the parser to deal with it, or we just reject it.

OR, as an alternative, we can have another token type that serves this specific purpose and catches all symbols that are known to be (somewhat) malformed and emit (or not) an error to warn the user. This would make Token only valid tokens.
^ update to that idea: it is not idea because we don't have a clear way to distinguish Token and such a Error token. Token is currently defined semantically as "not other tokens".

In fact, I don't know if trying a huge block is better or worse than trying each parser.

What do we do with invalid tokens? I think we should make them fail and not even lex. The question is whether we should lex `>/foo/` as `Token` and leave the parser to deal with it, or we just reject it. OR, as an alternative, we can have another token type that serves this specific purpose and catches all symbols that are known to be (somewhat) malformed and emit (or not) an error to warn the user. This would make `Token` only valid tokens. ^ update to that idea: it is not idea because we don't have a clear way to distinguish `Token` and such a Error token. `Token` is currently defined semantically as "not other tokens". In fact, I don't know if `try`ing a huge block is better or worse than `try`ing each parser.

primrose added 1 commit 2025-09-28 04:43:29 +00:00

ref(lexer): attempt to not try on every token ... e18082db1b

...for a better error message and better perf

primrose force-pushed reduce-try-usage from e18082db1b to 449b7c8ca7 2025-09-28 05:03:20 +00:00 Compare

elland added 1 commit 2025-09-28 19:55:21 +00:00

Added expression eval; adjusted birdtrack, added sol combinator. f5a9dec3d6

elland added 1 commit 2025-09-28 20:12:28 +00:00

Actually we don't care about start of doc, only line. 5beb321241

elland added 1 commit 2025-09-28 20:13:37 +00:00

delete sp file, some weird hls thing 63bd6841e8

elland force-pushed reduce-try-usage from 63bd6841e8 to f9423d4af0 2025-09-28 20:15:02 +00:00 Compare

elland commented 2025-09-29 10:20:26 +00:00

Owner

Copy link

I think for this lexer since it has to match haddock’s current markup we should just default to other and not fail or error. But we have the foundation for haddock 2 markup to be structured and stricter like Verso and give nice errors.

I think for this lexer since it has to match haddock’s current markup we should just default to other and not fail or error. But we have the foundation for haddock 2 markup to be structured and stricter like Verso and give nice errors.

👍 1

elland force-pushed reduce-try-usage from f9423d4af0 to 8c666f637c 2025-10-05 11:41:53 +00:00 Compare

elland added 1 commit 2025-10-05 12:02:00 +00:00

Added hls to shell.nix a36ff898e6

elland force-pushed reduce-try-usage from a36ff898e6 to a64ac93bd9 2025-10-05 15:20:34 +00:00 Compare

primrose commented 2025-10-06 13:36:32 +00:00

Author

Collaborator

Copy link

Maybe we shouldn't try to match \( and \) at lexer level? I don't think it's the lexer's responsibility to handle whether a construct is closed or not, lexer should only do character grouping.

Maybe we shouldn't try to match `\(` and `\)` at lexer level? I don't think it's the lexer's responsibility to handle whether a construct is closed or not, lexer should only do character grouping.

primrose commented 2025-10-21 03:51:54 +00:00

Author

Collaborator

Copy link

Update on this:

Current Haddock uses try everywhere too so we probably shouldn't care that much.

choice' :: [Parser a] -> Parser a
choice' [] = empty
choice' [p] = p
choice' (p : ps) = try p <|> choice' ps

Update on this: Current Haddock uses `try` everywhere too so we probably shouldn't care that much. ```hs choice' :: [Parser a] -> Parser a choice' [] = empty choice' [p] = p choice' (p : ps) = try p <|> choice' ps ```

👍 1

primrose closed this pull request 2025-10-21 03:52:00 +00:00

All checks were successful

Hide all checks

Haskell CI / build (pull_request) Successful in 2m51s

Details

Haskell CI / test (pull_request) Successful in 2m40s

Details

Haskell CI / fourmolu (pull_request) Successful in 6s

Details

Haskell CI / hlint (pull_request) Successful in 6s

Details

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: elland/haddock2#7

No description provided.