===== Crystal BBS - Part Two =====

==== Negotiating Telnet features with the client ====

OK, so now we have a basic Telnet hello world example going - let's move forward.

The plan for the tutorial at this stage is to handle some feature negotiation in this article, and the remaining negotiations and input on the next one. Haven't decided what to do after that, but probably a state machine interaction post or two, and then you should know most of what it takes to build a basic old-school BBS system. So, on to feature negotiation.

I've been using [[https://syncterm.bbsdev.net/|SyncTerm]] as my terminal problem for these posts. When connecting to the prototype BBS we're building, I noticed the feature negotiation flow using the "toggle options" telnet client trick from the [[tutorials:crystal_bbs:part_zero|intro post]]. We'll focus on a subset of options, and not the whole spectrum of possible options. IANA has [[http://www.iana.org/assignments/telnet-options/telnet-options.xhtml|a list]] of all Telnet options. After implementing these, others should hopefully be easy to implement as well.

Our prototype will:

  * show local echo
  * suppress Go-Ahead
  * ask for the user's terminal type
  * set the connection to binary-mode (so we can get input immediately without Enter keypresses)

All of this will be handled at the start of every connection, so let's revisit our connection handler method:

<code ruby>
private def handle_connection(socket)
  socket << "hello world\n"
  socket.close
end
</code>

Pretty bare at this moment. Let's start by telling the client we'll be handling feature negotiation. Also, to hold the connection info, let's introduce a new variable in the mix. 

<code ruby>
private def handle_connection(socket)
  socket << "detecting client features..."
  
  conn = ConnHash.new
  
  socket.close
end
</code>

We could write down the whole Crystal type, but since it's cumbersome we'll alias it to a shorter name (in this case, ''ConnHash''). To keep our main file uncluttered, let's move this to a separate file.

<code ruby>
module BBS
  alias ConnHash = Hash(Symbol, Bool | String | Int32)
end
</code>

Save this in the same folder as ''main.cr'' and name it ''aliases.cr'' since we'll be adding more aliases to this file as needed. Next, require it on ''main.cr'':

<code ruby>
require "./aliases"
</code>

The first option we want to negotiate is WILL ECHO to signal the client we want user input to be shown locally. That messag's IAC byte array will look like:

  * 1st byte - 0xff (255) / IAC
  * 2nd byte - 0xfb (251) / WILL
  * 3rd byte - 0x01 (  1) / ECHO

We could create the byte array "by hand" for each message, but let's make it a utility method:

<code ruby>
private def iac(*args)
  Slice(UInt8).new(args.size + 1) do |i| 
    i == 0 ? 255.to_u8 : args[i - 1].to_u8
  end
end
</code>

Looks weird, but it's quite simple. We start off by [[https://crystal-lang.org/docs/syntax_and_semantics/splats_and_tuples.html|splatting]] the arguments passed. This means we'll be able to handle a variable list of arguments, which serves us since some IAC messages have more arguments than others. We then create a byte array (''Slice(UInt8)'' in Crystal terms) with a size equal to the size of the arguments passed plus one - since we don't want to pass the IAC byte in every method call, we just insert it in the array, thus the plus one being added to the size, making room for it. 

In the block's main line, we use the ''i'' variable to loop through the array being constructed. When we're at the ''0'' index, we insert 255 (IAC) as an unsigned 8bit value, and if we're not at index ''0'', we just pass the arguments from the method into the byte array. Ingenious, and gets the job done. 

However, the ''255'' here does not make for very clean code, and we hate that, so let's create a file to house all of our constants. Save this as ''constants.cr'' in the same folder as our ''main.cr'' file:

<code ruby>
module BBS
  IAC  = 255
  
  DO   = 253
  DONT = 254
  
  WILL = 251
  WONT = 252
 
  SB   = 250
  SE   = 240

  BINARY        =  0
  ECHO          =  1
  SUP_GOAHEAD   =  3
  TERMINAL_TYPE = 24
end
</code>

As before, don't forget to require it in the main file:

<code ruby>
require "./constants"
</code>

With this file in place, we can now change the IAC method's source to something a bit more readable:

<code ruby>
private def iac(*args)
  Slice(UInt8).new(args.size + 1) do |i|
    i == 0 ? IAC.to_u8 : args[i - 1].to_u8
  end
end
</code>

Alright, with all this in place, let's abstract the negotiating to its own method on the main handler method:

<code ruby>
private def handle_connection(socket)
  socket << "detecting client features..."

  conn = ConnHash.new

  negotiate(iac(WILL, ECHO), :echo, socket, conn)

  socket.close
end
</code>

The ''negotiate'' method will take in a byte array (provided by our new ''iac'' utility method), a key ('':echo'' in this case), the ''socket'' and the ''conn'' (our hash to keep connection state). The key being passed is to be set on the ''conn'' hash as either true/false once the negotiation is done. The negotiate method looks like this:

<code ruby>
private def negotiate(slice, key, socket, conn)
  socket.write slice

  reply = Slice(UInt8).new(3)
  socket.read_fully reply

  case reply[1]
  when DO, WILL
    conn[key] = true
  when DONT, WONT
    conn[key] = false
  else
    puts "error negotiating #{key}!/#{reply.inspect}"
  end
end
</code>

First off, we write the slice (byte array) passed onto the socket. This will signal the client we want to initiate the negotiation for the feature encoded on the array. Then, we read the reply. The reply is always three bytes long, we so create a ''reply'' slice and read the response onto it from the socket. Then we get a bit lazy and only check the middle byte in the reply! Since the first byte will be IAC, and the last one is a repetition of the value we sent, we only care if the middle byte is DO/WILL or DONT/WONT.  Beware that in a proper system, we'd need some error handling here. Based on the yes/no character of the reply, we set the key passed on the ''conn'' hash as true or false.

Thus, after this method runs, the hash will hold if the client agreed to local echo or not! Let's add the remaining options, and print out a summary at the end:

<code ruby>
private def handle_connection(socket)
  socket << "detecting client features..."

  conn = ConnHash.new

  negotiate(iac(WILL, ECHO), :echo, socket, conn)
  negotiate(iac(WILL, SUP_GOAHEAD), :suppress_goahead, socket, conn)
  negotiate(iac(DO, BINARY), :binary, socket, conn)

  puts "features negotiated: #{conn}"

  socket.close
end
</code>

If you run ''make'' and connect to your local server with SyncTerm now, you should see:

<code>
/bbs_release
now listening in port 2023
features negotiated: {:echo => true, :suppress_goahead => true, :binary => true}
</code>

Great success! The client has agreed to all of our feature requests. There's one more request, but it requires special treatment. We want to get the "terminal type" which is a string that the terminal sends to identify itself. Of course, the reply being a string, it won't be a standard IAC 3-byte reply. The format of these special requests is:

  * 1st byte - 0xff / IAC (as usual)
  * 2nd byte - 0xfa / SB (special begin)
  * ... (request itself)
  * byte before last - 0xff / IAC
  * last byte - 0xf0 / SE (special end)

So, to make the client tell us its name, we'll send ''TERMINAL_TYPE''  ''ECHO'' from our constants (making the meat of the request ''0x18'' ''0x01'''). The response to this request will be:

  * 1st byte - IAC
  * 2nd byte - SB
  * 3rd byte - TERMINAL_TYPE (0x18)
  * 4th byte - IS (0x00)
  * .... (ANSI coded string)
  * byte before last - 0xff / IAC
  * last byte - 0xf0 / SE (special end)

Let's have another utility method to craft the IAC extended byte arrays:

<code ruby>
private def iac_sb(*args)
  Slice(UInt8).new(args.size + 4).tap do |slice|
    slice[0] = IAC.to_u8
    slice[1] = SB.to_u8
    args.each_with_index do |arg, idx|
      slice[idx + 2] = arg.to_u8
    end
    slice[-2] = IAC.to_u8
    slice[-1] = SE.to_u8
  end
end
</code>

This is pretty much the same logic as the ''iac'' method before, just making room for more predetermined bytes in the array being constructed. 

Now, to get the terminal string from the client:

<code ruby>
private def get_terminal_string(socket)
  socket.write iac_sb(TERMINAL_TYPE, ECHO)

  4.times { socket.read_byte }

  term_string = ""
  loop do
    this_byte = socket.read_byte
    if this_byte
      break if this_byte == IAC && socket.read_byte == SE
      term_string += this_byte.chr
    end
  end
  
  term_string
end
</code>

So we start off by sending the IAC extended request for the terminal type, using the ''iac_sb'' utility method. Then we skip the first four bytes (since we know they will be IAC / SB / TERMINAL_TYPE / IS) and start reading the string sent. 

We break if the byte being read is IAC and the next one is SE - we could just break on IAC, but by reading the next byte from the socket, we "empty" the bytes present on the socket's buffer. If the byte is not IAC, we add its ANSI value (using ''.chr'') to the terminal string, which is then returned.

Let's change the main handler method once more:

<code ruby>
private def handle_connection(socket)
  socket << "detecting client features..."

  conn = ConnHash.new

  negotiate(iac(WILL, ECHO), :echo, socket, conn)
  negotiate(iac(WILL, SUP_GOAHEAD), :suppress_goahead, socket, conn)
  negotiate(iac(DO, BINARY), :binary, socket, conn)

  negotiate(iac(DO, TERMINAL_TYPE), :terminal_type, socket, conn)
  conn[:term_string] = get_terminal_string(socket)

  puts "features negotiated: #{conn}"

  socket.close
end
</code>

You will see now if you connect to the server using SyncTerm, the server's output will show "ANSI" and if you use the command line ''telnet'' command, it will be "XTERM-256COLOR". Pretty exciting!

In the next part, we'll handle responding to features being requested by the client, and actual input. Stay tuned, see you next time!

Here's the full listing of our ''main.cr'' file so far:

<code ruby>
require "yaml"
require "socket"

require "./aliases"
require "./constants"

module BBS
  class Config
    YAML.mapping(
      settings: Hash(String, Int32)
    )
  end

  class Main
    @config : Config

    def initialize
      @config = load_config
    end

    def go!
      Signal::INT.trap { puts "SIGINT, exiting gracefully!"; exit 0 }

      telnet_port = @config.settings["port"]

      server = TCPServer.new("localhost", telnet_port)
      puts "now listening in port #{telnet_port}"
      loop do
        if socket = server.accept?
          spawn handle_connection(socket)
        end
      end
    end

    private def iac(*args)
      Slice(UInt8).new(args.size + 1) do |i|
        i == 0 ? IAC.to_u8 : args[i - 1].to_u8
      end
    end

    private def iac_sb(*args)
      Slice(UInt8).new(args.size + 4).tap do |slice|
        slice[0] = IAC.to_u8
        slice[1] = SB.to_u8
        args.each_with_index do |arg, idx|
          slice[idx + 2] = arg.to_u8
        end
        slice[-2] = IAC.to_u8
        slice[-1] = SE.to_u8
      end
    end

    private def negotiate(slice, key, socket, conn)
      socket.write slice

      reply = Slice(UInt8).new(3)
      socket.read_fully reply

      case reply[1]
      when DO, WILL
        conn[key] = true
      when DONT, WONT
        conn[key] = false
      else
        puts "error negotiating #{key}!/#{reply.inspect}"
      end
    end

    private def get_terminal_string(socket)
      socket.write iac_sb(TERMINAL_TYPE, ECHO)

      4.times { socket.read_byte }

      term_string = ""
      loop do
        this_byte = socket.read_byte
        if this_byte
          break if this_byte == IAC && socket.read_byte == SE
          term_string += this_byte.chr
        end
      end

      term_string
    end

    private def get_terminal_size(socket, conn)
      socket.write iac(DO, NAWS)

      reply = Slice(UInt8).new(9)
      socket.read_fully reply

      puts reply.inspect

      cols = reply[4]
      lines = reply[6]

      puts "NAWS -> cols: #{cols} lines: #{lines}"

      conn[:term_width] = cols.to_i
      conn[:term_height] = lines.to_i
    end

    private def handle_connection(socket)
      socket << "detecting client features..."

      conn = ConnHash.new

      negotiate(iac(WILL, ECHO), :echo, socket, conn)
      negotiate(iac(WILL, SUP_GOAHEAD), :suppress_goahead, socket, conn)
      negotiate(iac(DO, BINARY), :binary, socket, conn)

      negotiate(iac(DO, TERMINAL_TYPE), :terminal_type, socket, conn)
      conn[:term_string] = get_terminal_string(socket)

      get_terminal_size(socket, conn)

      puts "features negotiated: #{conn}"

      socket.close
    end

    private def load_config
      Config.from_yaml(File.read("config.yml"))
    end
  end
end
</code>