fix: fix NoMethodError when token_stream is nil

The change in #1055 might be a breaking change.
So, just simply wrap `token_stream` with `Array`

Co-authored-by: Jonathan Hefner <[email protected]>

Author: toshimaru

lib/rdoc/token_stream.rb

@@ -105,14 +105,14 @@ def pop_token
   # Current token stream
 
   def token_stream
-    @token_stream || []
+    @token_stream
   end
 
   ##
   # Returns a string representation of the token stream
 
   def tokens_to_s
-    token_stream.compact.map { |token| token[:text] }.join ''
+    Array(token_stream).compact.map { |token| token[:text] }.join ''
   end
 
 end

test/rdoc/test_rdoc_token_stream.rb

@@ -39,6 +39,13 @@ def test_class_to_html_empty
     assert_equal '', RDoc::TokenStream.to_html([])
   end
 
+  def test_token_stream
+    foo = Class.new do
+      include RDoc::TokenStream
+    end.new
+    assert_equal nil, foo.token_stream
+  end
+
   def test_tokens_to_s
     foo = Class.new do
       include RDoc::TokenStream