Crate scanlex

scanlex implements a simple lexical scanner.

Tokens are returned by repeatedly calling the get method, (which will return Token::End if no tokens are left) or by iterating over the scanner.

They represent floats (stored as f64), integers (as i64), characters, identifiers, or single or double quoted strings. There is also Token::Error to indicate a badly formed token. This lexical scanner makes some sensible assumptions, such as a number may not be directly followed by a letter, etc. No attempt is made in this version to decode C-style escape codes in strings. All whitespace is ignored.

Examples

use  scanlex::{Scanner,Token};

let mut scan = Scanner::new("iden 'string' * 10");
assert_eq!(scan.get(),Token::Iden("iden".into()));
assert_eq!(scan.get(),Token::Str("string".into()));
assert_eq!(scan.get(),Token::Char('*'));
assert_eq!(scan.get(),Token::Int(10));
assert_eq!(scan.get(),Token::End);

The scanner struct implements iterator, so:

let v: Vec<_> = scanlex::Scanner::new("bonzo 42 dog (cat)")
    .filter_map(|t| t.to_iden()).collect();
assert_eq!(v,&["bonzo","dog","cat"]);

Structs

• ScanError
  a scanner error type
• ScanLines
  used to generate Scanner structs for each line
• Scanner
  a struct for lexical scanning of a string

Enums

• Token
  Represents a token returned by Scanner::get