--- JSON-XS/XS.pm	2008/03/22 22:21:33	1.93
+++ JSON-XS/XS.pm	2008/03/25 07:46:15	1.94
@@ -683,6 +683,214 @@
 =back
 
 
+=head1 INCREMENTAL PARSING
+
+[This section is still EXPERIMENTAL]
+
+In some cases, there is the need for incremental parsing of JSON
+texts. While this module always has to keep both JSON text and resulting
+Perl data structure in memory at one time, it does allow you to parse a
+JSON stream incrementally. It does so by accumulating text until it has
+a full JSON object, which it then can decode. This process is similar to
+using C<decode_prefix> to see if a full JSON object is available, but is
+much more efficient (JSON::XS will only attempt to parse the JSON text
+once it is sure it has enough text to get a decisive result, using a very
+simple but truly incremental parser).
+
+The following two methods deal with this.
+
+=over 4
+
+=item [void, scalar or list context] = $json->incr_parse ([$string])
+
+This is the central parsing function. It can both append new text and
+extract objects from the stream accumulated so far (both of these
+functions are optional).
+
+If C<$string> is given, then this string is appended to the already
+existing JSON fragment stored in the C<$json> object.
+
+After that, if the function is called in void context, it will simply
+return without doing anything further. This can be used to add more text
+in as many chunks as you want.
+
+If the method is called in scalar context, then it will try to extract
+exactly I<one> JSON object. If that is successful, it will return this
+object, otherwise it will return C<undef>. This is the most common way of
+using the method.
+
+And finally, in list context, it will try to extract as many objects
+from the stream as it can find and return them, or the empty list
+otherwise. For this to work, there must be no separators between the JSON
+objects or arrays, instead they must be concatenated back-to-back.
+
+=item $lvalue_string = $json->incr_text
+
+This method returns the currently stored JSON fragment as an lvalue, that
+is, you can manipulate it. This I<only> works when a preceding call to
+C<incr_parse> in I<scalar context> successfully returned an object. Under
+all other circumstances you must not call this function (I mean it.
+although in simple tests it might actually work, it I<will> fail under
+real world conditions). As a special exception, you can also call this
+method before having parsed anything.
+
+This function is useful in two cases: a) finding the trailing text after a
+JSON object or b) parsing multiple JSON objects separated by non-JSON text
+(such as commas).
+
+=back
+
+=head2 LIMITATIONS
+
+All options that affect decoding are supported, except
+C<allow_nonref>. The reason for this is that it cannot be made to
+work sensibly: JSON objects and arrays are self-delimited, i.e. you can concatenate
+them back to back and still decode them perfectly. This does not hold true
+for JSON numbers, however.
+
+For example, is the string C<1> a single JSON number, or is it simply the
+start of C<12>? Or is C<12> a single JSON number, or the concatenation
+of C<1> and C<2>? In neither case you can tell, and this is why JSON::XS
+takes the conservative route and disallows this case.
+
+=head2 EXAMPLES
+
+Some examples will make all this clearer. First, a simple example that
+works similarly to C<decode_prefix>: We want to decode the JSON object at
+the start of a string and identify the portion after the JSON object:
+
+   my $text = "[1,2,3] hello";
+
+   my $json = new JSON::XS;
+
+   my $obj = $json->incr_parse ($text)
+      or die "expected JSON object or array at beginning of string";
+
+   my $tail = $json->incr_text;
+   # $tail now contains " hello"
+
+Easy, isn't it?
+
+Now for a more complicated example: Imagine a hypothetical protocol where
+you read some requests from a TCP stream, and each request is a JSON
+array, without any separation between them (in fact, it is often useful to
+use newlines as "separators", as these get interpreted as whitespace at
+the start of the JSON text, which makes it possible to test said protocol
+with C<telnet>...).
+
+Here is how you'd do it (it is trivial to write this in an event-based
+manner):
+
+   my $json = new JSON::XS;
+
+   # read some data from the socket
+   while (sysread $socket, my $buf, 4096) {
+
+      # split and decode as many requests as possible
+      for my $request ($json->incr_parse ($buf)) {
+         # act on the $request
+      }
+   }
+
+Another complicated example: Assume you have a string with JSON objects
+or arrays, all separated by (optional) comma characters (e.g. C<[1],[2],
+[3]>). To parse them, we have to skip the commas between the JSON texts,
+and here is where the lvalue-ness of C<incr_text> comes in useful:
+
+   my $text = "[1],[2], [3]";
+   my $json = new JSON::XS;
+
+   # void context, so no parsing done
+   $json->incr_parse ($text);
+
+   # now extract as many objects as possible. note the
+   # use of scalar context so incr_text can be called.
+   while (my $obj = $json->incr_parse) {
+      # do something with $obj
+
+      # now skip the optional comma
+      $json->incr_text =~ s/^ \s* , //x;
+   }
+
+Now lets go for a very complex example: Assume that you have a gigantic
+JSON array-of-objects, many gigabytes in size, and you want to parse it,
+but you cannot load it into memory fully (this has actually happened in
+the real world :).
+
+Well, you lost, you have to implement your own JSON parser. But JSON::XS
+can still help you: You implement a (very simple) array parser and let
+JSON decode the array elements, which are all full JSON objects on their
+own (this wouldn't work if the array elements could be JSON numbers, for
+example):
+
+   my $json = new JSON::XS;
+
+   # open the monster
+   open my $fh, "<bigfile.json"
+      or die "bigfile: $!";
+
+   # first parse the initial "["
+   for (;;) {
+      sysread $fh, my $buf, 65536
+         or die "read error: $!";
+      $json->incr_parse ($buf); # void context, so no parsing
+
+      # Exit the loop once we found and removed(!) the initial "[".
+      # In essence, we are (ab-)using the $json object as a simple scalar
+      # we append data to.
+      last if $json->incr_text =~ s/^ \s* \[ //x;
+   }
+
+   # now we have the skipped the initial "[", so continue
+   # parsing all the elements.
+   for (;;) {
+      # in this loop we read data until we got a single JSON object
+      for (;;) {
+         if (my $obj = $json->incr_parse) {
+            # do something with $obj
+            last;
+         }
+
+         # add more data
+         sysread $fh, my $buf, 65536
+            or die "read error: $!";
+         $json->incr_parse ($buf); # void context, so no parsing
+      }
+
+      # in this loop we read data until we either found and parsed the
+      # separating "," between elements, or the final "]"
+      for (;;) {
+         # first skip whitespace
+         $json->incr_text =~ s/^\s*//;
+
+         # if we find "]", we are done
+         if ($json->incr_text =~ s/^\]//) {
+            print "finished.\n";
+            exit;
+         }
+
+         # if we find ",", we can continue with the next element
+         if ($json->incr_text =~ s/^,//) {
+            last;
+         }
+
+         # if we find anything else, we have a parse error!
+         if (length $json->incr_text) {
+            die "parse error near ", $json->incr_text;
+         }
+
+         # else add more data
+         sysread $fh, my $buf, 65536
+            or die "read error: $!";
+         $json->incr_parse ($buf); # void context, so no parsing
+      }
+
+This is a complex example, but most of the complexity comes from the fact
+that we are trying to be correct (bear with me if I am wrong, I never ran
+the above example :).
+
+
+
 =head1 MAPPING
 
 This section describes how JSON::XS maps Perl values to JSON values and