Lasso Soft Inc. > Home

RhinoTrac

LassoSoft Ticket Tracking System

NOTE: If you are using Lasso Server 9.3 please Log your ticket directly via the LUX admin as this will give us more information about your issue.

Lasso 9 Issues



Ticket #7750: Encode_JSON slow

Reported by:
James Harvard
Date:
08 Apr, 2014
Priority:
major
Component:
Lasso 8.6
Version:
8.6.3
Keywords:
Platform:
All
Issue reported by James Harvard (james.harvard@harvard-digital.co.uk)
Company: Harvard Digital

Component: Lasso Pro 8.6
Version: 8.6.3
Platform: All
Source IP: 86.186.215.208

Detail
======
A reasonable quantity of JSON data (~125k) takes a long time (10-25+ seconds) to encode. The structure is:

<code>
map(
'key1' = map( /* small map */ ),
'key3' = array( /* small array */ ),
'key3' = array( /* large array of maps (database query result) */ )
)
</code>

I've rewritten the string encoding part of encode_JSON and tweaked the iteration through map and array types, which (for my string-heavy test-case) seems to run at least twice as fast now (circa 41% of the time taken to execute the original encode_JSON on the same data).

(I also removed from near the end of the code what looked to be a malformed duplicate of the first conditional clause that handles queues, lists, stacks and similar data types.)

The optimised version of the tag is below, which I'm happy to contribute back to the community, although if LassoSoft has some tests cases that you can check it against that would probably be prudent!

I don't know whether it's the recursive calling of the tag that is inherently slow in some way, but it would be ideal if LassoSoft could move this into faster code (C, Java, whatever - there seem to be lots of libraries listed at json.org).

<code>
Define_Tag: 'JSON', -Namespace='Encode_', -Required='value', -Optional='options';

Local( 'escapes' = Map('\\' = '\\', '"' = '"', '\r' = 'r', '\n' = 'n', '\t' = 't', '\f' = 'f', '\b' = 'b') );
Local: 'output' = '';
Local: 'newoptions' = (Array: -Internal);
If: !(Local_Defined: 'options') || (#options->(IsA: 'array') == False);
Local: 'options' = (Array);
/If;
If: (#options >> -UseNative) || (Params >> -UseNative);
#newoptions->(Insert: -UseNative);
/If;
If: (#options >> -NoNative) || (Params >> -NoNative);
#newoptions->(Insert: -NoNative);
/If;
If: (#options !>> -UseNative) && ((#value->(IsA: 'set')) || (#value->(IsA: 'list')) || (#value->(IsA: 'queue')) || (#value->(IsA: 'priorityqueue')) || (#value->(IsA: 'stack')));
#output += (Encode_JSON: Array->(insertfrom: #value->iterator) &, -Options=#newoptions);
Else: (#options !>> -UseNative) && (#value->(IsA: 'pair'));
#output += (Encode_JSON: (Array: #value->First, #value->Second));
Else: (#options !>> -Internal) && (#value->(Isa: 'array') == False) && (#value->(IsA: 'map') == False);
#output += '[' + (Encode_JSON: #value, -Options=#newoptions) + ']';
Else: (#value->(IsA: 'literal'));
#output += #value;
Else: (#value->(IsA: 'string'));
#output += '"';
local('char' = null, 'regexp' = regexp(-find='[\\x{0020}-\\x{21}\\x{23}-\\x{5b}\\x{5d}-\\x{10fff}]'));
iterate( #value, #char );
#output += (
#regexp->input(#char)&matches ? #char |
'\\' + (#escapes >> #char ? #escapes->find(#char) | 'u' + String(Encode_Hex(#char))->PadLeading(4, '0')&)
);
/iterate;
#output += '"';
Else: (#value->(IsA: 'integer')) || (#value->(IsA: 'decimal')) || (#value->(IsA: 'boolean'));
#output += (String: #value);
Else: (#value->(IsA: 'null'));
#output += 'null';
Else: (#value->(IsA: 'date'));
If: #value->gmt;
#output += '"' + #value->(format: '%QT%TZ') + '"';
Else;
#output += '"' + #value->(format: '%QT%T') + '"';
/If;
Else: (#value->(IsA: 'array'));
#output += '[';
local('temp');
iterate( #value, #temp );
#output += (Encode_JSON( #temp, -Options=#newoptions) + ', ');
/iterate;
#output->removetrailing(', ');
#output += ']';
Else: (#value->(IsA: 'object'));
#output += '{';
local('temp');
iterate( #value, #temp );
#output += (#temp->First + ': ' + Encode_JSON( #temp->Second, -Options=#newoptions) + ', ');
/iterate;
#output->removetrailing(', ');
#output += '}';
Else: (#value->(IsA: 'map'));
#output += '{';
local('temp');
iterate( #value, #temp );
#output += (Encode_JSON( #temp->First, -Options=#newoptions) + ': ' + Encode_JSON( #temp->Second, -Options=#newoptions) + ', ');
/iterate;
#output->removetrailing(', ');
#output += '}';
Else: (#value->(IsA: 'client_ip')) || (#value->(IsA: 'client_address'));
#output += (Encode_JSON: (String: #value), -Options=#newoptions);
Else: (#options !>> -NoNative);
#output += (Encode_JSON: (Map: '__jsonclass__'=(Array:'deserialize',(Array:'<LassoNativeType>' + #value->Serialize + '</LassoNativeType>'))));
/If;
Return: @#output;

/Define_Tag;
</code>

Comments

08 Apr, 2014 by James Harvard
BTW, a little note somewhere on how to format code nicely when submitting a bug report or comment would be great :-)
08 Jul, 2014 by Steve Piercy
FWIW, you might want to use Python for this.

The same issue affects Lasso 9.
http://www.stevepiercy.com/articles/lasso-theres-another-one-and-her-name-is-python/#json-de-encoding
Please log in to your LassoSoft account to comment

LassoSoft Inc. > Home

 

 

©LassoSoft Inc 2015 | Web Development by Treefrog Inc | PrivacyLegal terms and Shipping | Contact LassoSoft