RhinoTrac
LassoSoft Ticket Tracking System
NOTE: If you are using Lasso Server 9.3 please Log your ticket directly via the LUX admin as this will give us more information about your issue.
Ticket #7750: Encode_JSON slow
- Reported by:
- James Harvard
- Date:
- 08 Apr, 2014
- Priority:
- major
- Component:
- Lasso 8.6
- Version:
- 8.6.3
- Keywords:
- Platform:
- All
Issue reported by James Harvard (james.harvard@harvard-digital.co.uk)
Company: Harvard Digital
Component: Lasso Pro 8.6
Version: 8.6.3
Platform: All
Source IP: 86.186.215.208
Detail
======
A reasonable quantity of JSON data (~125k) takes a long time (10-25+ seconds) to encode. The structure is:
<code>
map(
'key1' = map( /* small map */ ),
'key3' = array( /* small array */ ),
'key3' = array( /* large array of maps (database query result) */ )
)
</code>
I've rewritten the string encoding part of encode_JSON and tweaked the iteration through map and array types, which (for my string-heavy test-case) seems to run at least twice as fast now (circa 41% of the time taken to execute the original encode_JSON on the same data).
(I also removed from near the end of the code what looked to be a malformed duplicate of the first conditional clause that handles queues, lists, stacks and similar data types.)
The optimised version of the tag is below, which I'm happy to contribute back to the community, although if LassoSoft has some tests cases that you can check it against that would probably be prudent!
I don't know whether it's the recursive calling of the tag that is inherently slow in some way, but it would be ideal if LassoSoft could move this into faster code (C, Java, whatever - there seem to be lots of libraries listed at json.org).
<code>
Define_Tag: 'JSON', -Namespace='Encode_', -Required='value', -Optional='options';
Local( 'escapes' = Map('\\' = '\\', '"' = '"', '\r' = 'r', '\n' = 'n', '\t' = 't', '\f' = 'f', '\b' = 'b') );
Local: 'output' = '';
Local: 'newoptions' = (Array: -Internal);
If: !(Local_Defined: 'options') || (#options->(IsA: 'array') == False);
Local: 'options' = (Array);
/If;
If: (#options >> -UseNative) || (Params >> -UseNative);
#newoptions->(Insert: -UseNative);
/If;
If: (#options >> -NoNative) || (Params >> -NoNative);
#newoptions->(Insert: -NoNative);
/If;
If: (#options !>> -UseNative) && ((#value->(IsA: 'set')) || (#value->(IsA: 'list')) || (#value->(IsA: 'queue')) || (#value->(IsA: 'priorityqueue')) || (#value->(IsA: 'stack')));
#output += (Encode_JSON: Array->(insertfrom: #value->iterator) &, -Options=#newoptions);
Else: (#options !>> -UseNative) && (#value->(IsA: 'pair'));
#output += (Encode_JSON: (Array: #value->First, #value->Second));
Else: (#options !>> -Internal) && (#value->(Isa: 'array') == False) && (#value->(IsA: 'map') == False);
#output += '[' + (Encode_JSON: #value, -Options=#newoptions) + ']';
Else: (#value->(IsA: 'literal'));
#output += #value;
Else: (#value->(IsA: 'string'));
#output += '"';
local('char' = null, 'regexp' = regexp(-find='[\\x{0020}-\\x{21}\\x{23}-\\x{5b}\\x{5d}-\\x{10fff}]'));
iterate( #value, #char );
#output += (
#regexp->input(#char)&matches ? #char |
'\\' + (#escapes >> #char ? #escapes->find(#char) | 'u' + String(Encode_Hex(#char))->PadLeading(4, '0')&)
);
/iterate;
#output += '"';
Else: (#value->(IsA: 'integer')) || (#value->(IsA: 'decimal')) || (#value->(IsA: 'boolean'));
#output += (String: #value);
Else: (#value->(IsA: 'null'));
#output += 'null';
Else: (#value->(IsA: 'date'));
If: #value->gmt;
#output += '"' + #value->(format: '%QT%TZ') + '"';
Else;
#output += '"' + #value->(format: '%QT%T') + '"';
/If;
Else: (#value->(IsA: 'array'));
#output += '[';
local('temp');
iterate( #value, #temp );
#output += (Encode_JSON( #temp, -Options=#newoptions) + ', ');
/iterate;
#output->removetrailing(', ');
#output += ']';
Else: (#value->(IsA: 'object'));
#output += '{';
local('temp');
iterate( #value, #temp );
#output += (#temp->First + ': ' + Encode_JSON( #temp->Second, -Options=#newoptions) + ', ');
/iterate;
#output->removetrailing(', ');
#output += '}';
Else: (#value->(IsA: 'map'));
#output += '{';
local('temp');
iterate( #value, #temp );
#output += (Encode_JSON( #temp->First, -Options=#newoptions) + ': ' + Encode_JSON( #temp->Second, -Options=#newoptions) + ', ');
/iterate;
#output->removetrailing(', ');
#output += '}';
Else: (#value->(IsA: 'client_ip')) || (#value->(IsA: 'client_address'));
#output += (Encode_JSON: (String: #value), -Options=#newoptions);
Else: (#options !>> -NoNative);
#output += (Encode_JSON: (Map: '__jsonclass__'=(Array:'deserialize',(Array:'<LassoNativeType>' + #value->Serialize + '</LassoNativeType>'))));
/If;
Return: @#output;
/Define_Tag;
</code>
Please log in to your LassoSoft account to comment
Comments
The same issue affects Lasso 9.
http://www.stevepiercy.com/articles/lasso-theres-another-one-and-her-name-is-python/#json-de-encoding