This package implements a grammar specification and tools to compile the grammar and use it for driving CLIs. The CLI commands can be expressed in grammar with the support for custom tokens with suggestion and autocompletion.

The grammar specification is parsed with PLY python package and the CLI objects are implemented on top of readline. The command loop is in async mode.

asyncio: The commandline interpreter will be looping for console input in async mode

ply: The lex-yacc like implementation of python. Will be used for parsing the grammar specification and tokenizing the line input

nessaid_readline: For common line reading and editing support across Windows, Linux and MAc platforms

This package provides two classes for CLI building. NessaidCmd is intended to work like the standard Cmd class for simple CLI designs. It is a stripped down sub class of the more robest NessaidCli class. For the initial release only NessaidCmd is being documented here.

NessaidCmd: The basic Cmd like tool for end user. The CLI command handlers are defined as python methods with chosen prefix and the grammar definitions to drive them are expressed as the method docstring.

NessaidCli: The generic base class for the CLI impelemntation. NessaidCmd is the stripped down version as an alternative for the default Cmd implementation

This package can be installed by cloning this repo and running python3 setup.py install from top directoy. It's also available in pypi. pip3 install nessaid_cli will install the package.

Check the example CLIs built in doc directory which has updated API usage and features. The step by step process of composing a CLI tool is explained in the router-box example

The grammar used to define the CLI commands is a simple context free grammar. The grammar specification can have definitions for individual tokens and multiple named grammars composed of the tokens.

Tokens are terminal symbols which are passed to the parser as strings. By default the CLI framework splits the input sequence with space as delimiter. If we use quoted string, we can have spaces inbetween words of a single token. ie. token1, "token2", "token 3" are 3 possible inputs for tokens

The following sub sections gives more details on token types

In the grammar we can express a basic token by either quoted string which will be used as a token matching to itself.

Or we can have a token definition like the following

token tokenx;

which will be used in the CLI as a token matching the string "tokenx"

We can specify the tokens by the following expressions

token token1 TokenClass1();
token token2 TokenClass2(arg1, arg2);

In this case token definitions will be generated by initializing objects of the classes with optional parameters. For example,

token str_token StringToken();

will generate a token which will match any string from the input

token integer RangedIntToken(-100, 200, 5);

will match any integer between -100 and 200. The third parameter in the above example is an extra argument we pass to limit the number of suggestion. If we add the token like this and type 2 and TAB from CLI, sine the number of possible matches are more than 5, ie 2, 20-29 ,and 200, the CLI will print the suggestion as "An integer between -100 and 200" If we type 20 and TAB we have only 2 possible completions, 20 and 200 so the CLI will give suggestion 20 and 200

NOTE: If we use token classes, we should override the get_token_classes method of NessaidCmd class so as to return the token classes used for the CLI. See examples in the end.

We can define all the tokens at the start of the CLI specification. And later we can use them in grammar definitions.

token token1;
token token2;
token str_token StringToken();
token integer RangedIntToken(-100, 200, 5);

Please refer to README.tokens.md in the doc folder

From the above tokens we can build a simple grammar like below

test_grammar:
    "test-string-constant-token"
    token1
    token2
    ;

This will match the only possible input

test-string-constant-token token1 token2

Now if we give the following definition

test_grammar2:
    str_token integer
    ;

it will match any string followed by an integer between -100 and 200

Any token or combination of tokens can be made optional by enclosing them in braces. For example modifying the above example,

test_grammar:
    "test-string-constant-token"
    { token1 }
    token2
    ;

will match the following possible inputs

test-string-constant-token token1 token2
test-string-constant-token token2

Enclosing the tokens or combinations in parentheses will make it act as a single block when combining with other grammar blocks

We can specify alternatives of tokens and combinations by the | (OR) symbol eg.

test_grammar:
    "test-string-constant-token"
    |
    (
        token1
        token2
    )
    ;

Will match either of the following

token1 token2
test-string-constant-token

Any grammar block can be suffixed with a multiplier to match it more than once

test_grammar:
    (
        token1
        token2
    ) * 2
    ;

will match

token1 token2 token1 token2

test_grammar:
    "test-string-constant-token" * (1:3)
    ;

will match minimum 1 and maximum 3 repetitions of test-string-constant-token

If we want to match a number of elements in any order, the set expression can be used. The set expression is as follows

expr1, expr2, expr3

The above expression will match the 3 expressions in any order. One thing to be noted is, set expressions should be enclosed in parentheses (for mandatory matches) or braces (for optional matches)

for example

(
    "token1",
    "token2"
    "token3"
)

will match any combination of the three tokens.

{
    "token1",
    {
        "token2"
    },
    "token3"
}

will match the same except that token2 can be omitted as it is optional and the whole block can be omitted as that is also optional

The grammars can be referenced by their name from other grammars too, like normal tokens. See the example.

grammar1[$var]:
    "grammar1"
    <<$var = "From grammar 1";>>
    ;

grammar2[$var]:
    "grammar2"
    <<$var = $1;>>
    ;

main_grammar[]:
    <<
        $msg = "";
    >>
    (
        grammar1[$msg]
        |
        grammar2[$msg]
    )
    <<print($msg);>>
    ;

If we run the grammar main_grammar from CLI, it will match either of the strings "grammar1" or "grammar2" and because main_grammar passes it's local variable $msg to the sub grammars, they will fill the variable and when the print statement hits, we can see different messages getting printed according to which token we input. ie either "From grammar 1" or "grammar2" will get printed.

We can specify the actions to be executed when we match parts or the entire grammar. The actions are statements enclosed in
<< and >> These actions will let us bind the grammar with our CLI and do operations based on grammar match

The grammar can incorporate the following types of variables

Sample usage

test_grammar[$param1, $param2]:
    <<
    print("Incoming params:", $param1, $param2);
    >>
    (
        "test-string-constant-token" * (1:3)
    )
    <<
        $param1 = 1;
        $param2 = 2;
    >>
    ;

If the grammar matches, the arguments passed to the grammar will be printed from the initial action defined. print there is an internal function call supported for the grammar actions. Also later the values of the parameters will be changed to 1 and 2 respectively.

Sample usage

test_grammar[$param1, $param2]:
    <<
    $var1 = 1;
    $var2 = "Two";
    $var3 = list();
    $var4 = dict();
    $var5 = set();
    >>
    (
        "test-string-constant-token" * (1:3)
    )
    <<
        $var5 = append($var5, 1);
        $var4 = update($var4, "key", "value");
    >>
    ;

The input values for tokens can be extracted using token variables. A sequence of terminal tokens will be numbered 1 to n and $1 to $n will hold the matched values For example in the following grammar

test_grammar[$var1]:
    str_token <<$var1 = $1;>>
    integer << $var2 = $2;>>
    {
        integer << $var3 = $1; >>
        "test-string-constant-token" <<print($2);>>
    }
    ;

The statement

str_token <<$var1 = $1;>>

Will work like this. The $1 expression will extract the string input (str_token is defined to match any string) and change the value of $var1, which is a variable passed to the grammar from outer levels. This inturn will modify the value of the variable in called context

integer << $var2 = $2;>>

This will create a local variable $var2 and set the value to the matched integer

integer << $var3 = $1; >>

A new variable. As this is inside a new optional block, this is a new sequence, so $1 will be used to hold the value

"test-string-constant-token" <<print($2);>>

This will print the string "test-string-constant-token" to console

As part of the grammar actions, we can either call functions internal to the CLI framework or designated hook functions. Here are the inline functions supported

Works similar to python's print method. Accepts a number of printable arguments and retuns None

Creates a list object with optional arguments and returns the list

Accepts no arguments and returns an empty dict. Can be filled with the update function call

Creates a set object with optional arguments and returns the set

Accepts at least two arguments. First argument is a list or set. The subsequent arguments are added to it and returned.

Accepts 3 arguments. 1st one is a dict the second one is key and 3rd one value. The dist is updated with {key: value}

Accepts n arguments and retuns the sum

Mathematical operations. accepts one argument. Argument should be a number. returns arg + 1 or arg -1 respectively

To call external functions, we should define them in the Cli/Cmd class. Then we can use it the following way.

$var = call func1();
call $func2(arg1, arg2);

We can override the get_cli_hook method to control the functions designated for calling from grammar. The method accepts the name of the function used in grammar and returns the actual method in the object which should be called.

Comments can be added either in grammars portions or in action blocks Single line comments can be done using either '#' (Hash) or "//" (Double forward slash) Multi line comments can be done by enclosing the text between /* and */

Here's a basic example of running a custom grammar on CLI. However the initial release intend to focus on the Cmd utility, which automates most of the steps. Here's the basic usage

The python code test.py

import os
from nessaid_cli.compiler import compile_grammar
from nessaid_cli.cli import NessaidCli

from nessaid_cli.tokens import (
    RangedIntToken,
    RangedStringToken,
    AlternativeStringsToken,
)

class TestCli(NessaidCli):

    def get_token_classes(self):
        """Method to override.
        It should return the list of token classes being used"""
        return [RangedIntToken, RangedStringToken, AlternativeStringsToken]

    def exit(self):
        """This will be called from exit command of the CLI grammar"""
        self.exit_loop()


if __name__ == '__main__':

    grammar_file = os.path.join(os.path.dirname(__file__), "test_input.g")

    with open(grammar_file) as fd:
        inp_str = fd.read()
        grammar_set = compile_grammar(inp_str)

    cli = TestCli(grammar_set, prompt="# ")
    # 'test_grammar' is the grammar to load with the CLI, part of test_input.g
    cli.run('test_grammar')

The grammar file test_input.g

token TEST_NUMBER RangedIntToken(1, 100); // Token to match integer between 1 and 100
token STRING_TOKEN RangedStringToken(5, 10); // Token to match a string of length 5 to 10

command1[$number_var]:
	"command1"
	TEST_NUMBER
	<<
		print("Incoming number is:", $number_var); # This uses inline print function
		call print("Input number is:", $2); /* Note call prefix. So routed to the CLI class's print function */
	>>
	;

command2[$str_var]:
	"command2"
	STRING_TOKEN
	<<
		print("Incoming str is:", $str_var);
		call print("Input str is:", $2);
	>>
	;

command3:
	"command3"
	<< print("This is simply command3"); >>
	;

exit:
	("exit" | "quit")
	<<call exit();>>
	;

test_grammar:
	<<
		$number = 5;
		$msg = "This is test_grammar";
	>>
	command1[$number]
	|
	command2[$msg]
	|
	command3
	|
	exit
	;

The CLI session (Windows, <ENTER> and <TAB> means Enter and Tab key presses)

python test.py
Starting Nessaid CMD Demo
# <ENTER>
# <TAB>
command1
command2
command3
# command1 <TAB>
An integer between 1 and 100
# command1 101 <ENTER>
Result: failure
Error: Could not match any rule for this sequence
# command1 100  <ENTER>
Incoming number is: 5
External function: Input number is: 100
# comm <TAB>
command1
command2
command3
# command2 <TAB>
Any string of length (5-10)
# command2 asdf <ENTER>
Result: failure
Error: Could not match any rule for this sequence
# command2 asdfg <ENTER>
Incoming str is: This is test_grammar
External function: Input str is: asdfg
# command2 asdfgsSsSSS <ENTER>
Result: failure
Error: Could not match any rule for this sequence
# comm <TAB>
command1
command2
command3
# command3 <ENTER>
This is simply command3
#

Some of the steps in above CLI implementation is automated in NessaidCmd class, which is stripped down to be used like pythons cmd package. The Cmd loop is in async mode. Here we dont have to define each grammar, instead we will add the global token and shared grammar definitions as part of the class docstring. We will define methods in our class and grammar for each command will be given as the docstring for the method. When the grammar in the docstring matches, the corresponding method will be invoked. The methods designated as Cmd command handlers should be named with a common prefix. do_ is the default prefix used in the class. Now let's try to implement the above CLI ecample using NessaidCmd class. Note that here we need only the python file. The grammar will be docstrings

from nessaid_cli.cmd import NessaidCmd

from nessaid_cli.tokens import (
    RangedIntToken,
    RangedStringToken
)

class TestCmd(NessaidCmd):
    r"""
    token TEST_NUMBER RangedIntToken(1, 100); // Token to match integer between 1 and 100
    token STRING_TOKEN RangedStringToken(5, 10); // Token to match a string of length 5 to 10
    """

    def get_token_classes(self):
        """Method to override.
        It should return the list of token classes being used"""
        return [RangedIntToken, RangedStringToken]

    def do_command1(self, number):
        r"""
        "command\n1"
        TEST_NUMBER
        <<
            $number = $2;
        >>
        """

        """
        The Cmd framework here does the following

        1. Generate a named grammar corresponding to the do_command1 method
            with the method's docstring as grammar body.
        2. The grammar body will have local variables generated for each parameter
            of the method. ie here a local variable $number will be available for the
            grammar so that we can assign the input number to it in the action.
            If we don't assign anything the value of the variable will be ""
        3. Generate a master grammar which is the alternatives of the method grammars
            ie. Like

            master_grammar:
                do_command1
                |
                do_command2
                |
                do_command3
                ;

        4. When the grammar of a method matches the method will be called with the
            arguments.
        """

        # Now that the grammar matched, the function will be called with the arguments
        # we prepared. Process them in the function
        print("Incoming variable is not there. The feature is not yet ready for Cmd. This is the python print method")
        print("Input number is:", number)

    def do_command2(self, string):
        r"""
        "command2"
        STRING_TOKEN
        <<
            $string = $2;
            print("Inline print: Input number:", $2);
            call print("Input number:", $2);
        >>
        """
        print("Input str is:", string);

    def do_command3(self, string, number, numbers):
        r"""
        <<
            $numbers = list();
        >>

        "command3"
        STRING_TOKEN
        <<
            $string = $2;
        >>
        TEST_NUMBER
        <<
            $number = $3;
        >>
        # Now we may input an optional list with minimum 1 and maximum 3 integers
        {
            (
                TEST_NUMBER << $numbers = append($numbers, $1);>>
            ) * (1:3)
        }
        """
        print("Input str is:", string);
        print("Input number is:", number);
        print("Input list is:", numbers);


if __name__ == '__main__':
    cmd = TestCmd(prompt="nessaid-cmd # ", show_grammar=True)
    #show_grammar will print the generated grammar specification
    cmd.run(intro="Starting Nessaid CMD Demo")

Now let's see the interactive CLI. The commands 'exit' 'quit' and 'end' are handled by base class with the method do__exit method to end the CLI session. We may override the function to disable, reconfigure it.

python test.py
# Generated CLI grammar:

token TEST_NUMBER RangedIntToken(1, 100); // Token to match integer between 1 and 100
token STRING_TOKEN RangedStringToken(5, 10); // Token to match a string of length 5 to 10


do__exit[$arg_1, $arg_2, $arg_3]:
  (
    "exit" | "quit" | "end"

  )
  <<call do__exit();>>
  ;

do_command1[$arg_1, $arg_2, $arg_3]:
  <<
  $number = "";
  >>
  (
    "command1"
    TEST_NUMBER
    <<
        $number = $2;
    >>

  )
  <<call do_command1($number);>>
  ;

do_command2[$arg_1, $arg_2, $arg_3]:
  <<
  $string = "";
  >>
  (
    "command2"
    STRING_TOKEN
    <<
        $string = $2;
        print("Inline print: Input number:", $2);
        call print("Input number:", $2);
    >>

  )
  <<call do_command2($string);>>
  ;

do_command3[$arg_1, $arg_2, $arg_3]:
  <<
  $string = "";
  $number = "";
  $numbers = "";
  >>
  (
    <<
        $numbers = list();
    >>

    "command3"
    STRING_TOKEN
    <<
        $string = $2;
    >>
    TEST_NUMBER
    <<
        $number = $3;
    >>
    # Now we may input an optional list with minimum 1 and maximum 3 integers
    {
        (
            TEST_NUMBER << $numbers = append($numbers, $1);>>
        ) * (1:3)
    }

  )
  <<call do_command3($string, $number, $numbers);>>
  ;

TestCmd[]:
  do__exit
  |
  do_command1
  |
  do_command2
  |
  do_command3
  ;

Starting Nessaid CMD Demo
nessaid-cmd # <TAB>
command1
command2
command3
end
exit
quit
nessaid-cmd # c <TAB>
command1
command2
command3
nessaid-cmd # command <TAB>
command1
command2
command3
nessaid-cmd # command1 <TAB>
An integer between 1 and 100
nessaid-cmd # command1 55 <TAB>
NEWLINE: Complete command
nessaid-cmd # command1 55 <ENTER>
Incoming variable is not there. The feature is not yet ready for Cmd. This is the python print method
Input number is: 55
nessaid-cmd # comm <TAB>
command1
command2
command3
nessaid-cmd # command2 <TAB>
Any string of length (5-10)
nessaid-cmd # command2 as <ENTER>
Result: failure
Error: Could not match any rule for this sequence
nessaid-cmd # command2 asdfgh <ENTER>
Inline print: Input number: asdfgh
External function: Input number: asdfgh
Input str is: asdfgh
nessaid-cmd # command2 asdfghhhhhhh <ENTER>
Result: failure
Error: Could not match any rule for this sequence
nessaid-cmd # com <TAB>
command1
command2
command3
nessaid-cmd # command3 <TAB>
Any string of length (5-10)
nessaid-cmd # command3 12345
An integer between 1 and 100
nessaid-cmd # command3 12345 20 <TAB>
An integer between 1 and 100
NEWLINE: Complete command
nessaid-cmd # command3 12345 20 10 <TAB>
10
100
nessaid-cmd # command3 12345 20 10 <TAB>
An integer between 1 and 100
NEWLINE: Complete command
nessaid-cmd # command3 12345 20 10 11 <TAB>
Input str is: 12345
Input number is: 20
Input list is: [10, 11]
nessaid-cmd # command3 12345 20 10 11 <TAB>
An integer between 1 and 100
NEWLINE: Complete command
nessaid-cmd # command3 12345 20 10 11 30 <TAB>
NEWLINE: Complete command
nessaid-cmd # command3 12345 20 10 11 30 <ENTER>
Input str is: 12345
Input number is: 20
Input list is: [10, 11, 30]
nessaid-cmd # command3 12345 20 10 11 30 40 <TAB>
Failed to parse: Could not match any rule for this sequence
nessaid-cmd # command3 12345 20 10 11 30 40 <ENTER>
Result: failure
Error: Could not match any rule for this sequence
nessaid-cmd # <ENTER>
nessaid-cmd # command3 12345 20 <ENTER>
Input str is: 12345
Input number is: 20
Input list is: []
nessaid-cmd # command3 12345 20 15 <ENTER>
Input str is: 12345
Input number is: 20
Input list is: [15]
nessaid-cmd # e <TAB>
end
exit
nessaid-cmd # ex <ENTER>

Note the parameters in the generated method grammars. They are dummy as of now.

Supports following basic line editing options

Arrow keys History and search CTRL+R for backward lookup, CTRL+S for forward lookup PAGE_UP for first history entry page down for last

HOME or ATRL+A for beginning of line END or CTRL+E for end of line

INSERT for toggling INSERT/REPLACE

CTRL+C for canceling current input CTRL+D to exit CLI

An example with a sigle cli object is given in the doc directory (simple_cli.py). The example inplementation showcases how to match basic token types with auto completion and how to define new token classes

A working example of a minimal router-box CLI is in the doc directory. This include sub cli contexts which are used to configure a sub section of our configuration.

Once we define our command grammar in Cmd, we can invoke those commands by passing the arguments to the python executable Basic usage is like

# file cmd_class.py

class CmdClass(NessaidCmd):
    # Class definition


if __name__ == '__main__':
    if len(sys.argv) > 1:
        args = sys.argv[1:]
        sys.exit(CmdClass.execute_args(*args))
        # or sys.exit(CommandLineCmd().exec_args(*args))

where CmdClass is a proper Cmd implementation with commands. If CmdClass has a command 'command1' which accepts , it can be run from CLI like

python[3] cmd_class.py command1 <args>

Check the examples in examples/command_line_args_test.py code and follow up CLI output

• Support for importing other grammar files. The parser supports import statements but processing is not there.
• Add some kind of privilege states for tokens so that tokens will be matched only if the privilege level of the CLI instance is higher than that of the token.
• Compiler to generate a json or similar format for compiled grammar so that it can be loaded to CLI processors. Now the python parsed tree is used. If we can convert between json and the parse tree, I think it will be good to use across different languages as implementation of the compiler might be tedious in all languages.
• A proper way to feed variables to Cmd method grammars from calling python code
• Expensive validation and checks for recursive grammars. Basic recursive grammars are tested.
• Enhancement of the parsers to track the syntax and parsing errors clearly with line and column numbers. The current implementation on the part is partial. Sometimes, tough to track the grammar mistakes.
• Support for more inline functions and types. May be a way to register inline functions.