May I ask how you tracked it down? For example, what clues or
symptoms led you to discover the root cause? What techniques
did you use, e.g., watch network traffic, logs?

In Lasso 8.x, one could set SiteAdmin to ignore the -host
parameter in email_send.

--steve


On 8/15/12 at 2:39 PM, lasso@totallymac.com (Daniel Parker) pronounced:

>>>
>>>> Lasso 9.2.3 64bit CentOS 6.
>>>> Hi All,
>>>> After thinking that my CPU at 100% was caused by external
>>>>sessions, it now
>looks like it wasn't, sorry :-) ..
>>>> The issue is intermittently showing up on our production
>>>>server, but not on
>our development machines.
>>>> Two questions:
>>>> 1) This is a quad core server, and when lasso is at 100%
>>>>CPU, it is using 100%
>of one core.. the other 3 cores are mostly idle. Lasso is
>completely unresponsive to access via HTTP. The question is,
>how do I get lasso to spawn more threads for each core? (is
>this even possible?)
>>>> 2) Im thinking the issue could be caused by an infinite
>>>>loop in our code, or
>something silly like that. It 8.5.6 you could set a recursion
>limit or a thread time limit.. how do you do this in 9?
>>>> This is starting to stress me a little.. :-)
>>>> Daniel Parker
>
>OK.. The issue was that I was using a variable as the -Host in
>an Email_Send tag.. This variable was null (unintentionally)..
>Lasso will then consume 100% CPU which is caused by a stuck
>message in the SQLite smtp_queue DB. Upon restart lasso will be
>at 100% CPU within 1 minute.
>
>The Fix is to delete the contents of this table.
>
>Our particular table contents looked like this:
>
>.dump smtp_queue
>PRAGMA foreign_keys=OFF;
>BEGIN TRANSACTION;
>CREATE TABLE smtp_queue (id INTEGER, id_site INTEGER, id_route
>INTEGER, id_cache INTEGER, mime_text TEXT, mime_recipients
>TEXT, mime_from TEXT, smtp_host TEXT, smtp_port INTEGER DEFAULT
>25, smtp_timeout INTEGER DEFAULT 15, smtp_username TEXT,
>smtp_password TEXT, smtp_ssl TEXT, queue_date TEXT,
>queue_status TEXT, queue_error TEXT, queue_number INTEGER
>DEFAULT 0, queue_priority INTEGER DEFAULT 0, queue_retries
>INTEGER DEFAULT 0, merge_array TEXT, merge_tokens TEXT,
>create_date TEXT, PRIMARY KEY (id) );
>INSERT INTO "smtp_queue" VALUES(1,'63a28bf1-d7f6-4d90-b63d-d8833eecaca2',NULL,NULL,X'
>4D494D452D56657273696F6E3A20312E300D0A582D4D61696C65723A204C6173736F2050726F666573
>73696F6E616C20392E320D0A4D6573736167652D49443A203C35373964613364622D636661612D3463
>62372D626363642D633663333633343461303938403132372E302E302E313E0D0A446174653A205765
>642C2031352041756720323031322031323A30333A3137202B313230300D0A546F3A206C6F67734062
>65657263656C6C61722E636F2E6E7A0D0A46726F6D3A2062756773406265657263656C6C61722E636F
>6D0D0A5375626A6563743A204E657720427567205265706F727465640D0A436F6E74656E742D547970
>653A20746578742F706C61696E3B20636861727365743D7574662D380D0A436F6E74656E742D547261
>6E736665722D456E636F64696E673A2071756F7465642D7072696E7461626C650D0A0D0A200D0A','
>logs@example.com','bugs@example.com','','','','','','','2012-08-15
>12:03:17','queued','',0,3,0,X'6E756C6C',X'6E756C6C','2012-08-15 12:03:17');
>COMMIT;
>
>I feel that this was my own stupid fault, but maybe Lasso
>should't completely kill itself in this situation.
>
>Im posting this to the list as I hope it helps someone else in the future.
>
>Daniel Parker
>
>#############################################################
>This message is sent to you because you are subscribed to
>the mailing list Lasso
>Lasso@lists.lassosoft.com
>To unsubscribe, E-mail to: <Lasso-unsubscribe@lists.lassosoft.com>
>Send administrative queries to <Lasso-request@lists.lassosoft.com>

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- --
Steve Piercy Web Site Builder
Soquel, CA
<web@StevePiercy.com> <http://www.StevePiercy.com/>

#############################################################
This message is sent to you because you are subscribed to
the mailing list Lasso
Lasso@lists.lassosoft.com
To unsubscribe, E-mail to: <Lasso-unsubscribe@lists.lassosoft.com>
Send administrative queries to <Lasso-request@lists.lassosoft.com>

On 15/08/2012, at 4:03 PM, Steve Piercy - Web Site Builder wrote:

> May I ask how you tracked it down? For example, what clues or symptoms led you to discover the root cause? What techniques did you use, e.g., watch network traffic, logs?
>
> In Lasso 8.x, one could set SiteAdmin to ignore the -host parameter in email_send.

A whole lot of pain.. Initially I thought it was external sessions, so we swapped back to internal (on a three week old backup of the VM) and the problem went away for about 96 hours.

When it came back, it would occur within 1 minute of lasso loading, prior to it service any request. Apache would still serve files when Lasso was in a locked up state.

Thinking it was session related, I went hunting through any log file I could find.. In Lasso.out.txt I had the following:

Failure in sqlite_session_driver active_tick: Error from SQLite database "lasso_session": 19 constraint failed
[2012-08-14 16:47:41] SMTP: [22] Sending messages to host 127.0.0.1.
[2012-08-14 16:47:41] SMTP: [22] Done sending messages.

So I thought that it sounded bad, and started digging in sqllite.. first I dumped the session db to see if it was a bad session, no luck.

Then I started looking at each email db.. and saw there was a message in the smtp_queue..
So I tried to dump it:

delete from smtp_queue;
Error: database is locked

It was locked because Lasso was going nuts on it.. Stopping Lasso allowed me to delete it, and then Lasso has worked ever since.

Looking through the dump (in the previous post) I could see that smtp_host was null. This is usually defined by a variable, so I created a test file on our development machine that tried to do an email_send with a null host, and the same issue occurred.

Some of this was logical, and some of it was luck, but it is now fixed in my code, and hopefully can be addressed in a Lasso update (I still believe it should either error completely, or fail gracefully)

Daniel Parker


#############################################################
This message is sent to you because you are subscribed to
the mailing list Lasso
Lasso@lists.lassosoft.com
To unsubscribe, E-mail to: <Lasso-unsubscribe@lists.lassosoft.com>
Send administrative queries to <Lasso-request@lists.lassosoft.com>

Thank you for sharing! That's some fine investigative work.

I've added this as a note in my troubleshooting checklist when
8.x crashes as well.

I'm seeing daily restarts of 8.6's default site/instance in my
logs, and most of the restarts go through without a hitch. But
every once in a while, the default-1 site for 8.6 will not restart.

--steve


On 8/15/12 at 4:49 PM, lasso@totallymac.com (Daniel Parker) pronounced:

>On 15/08/2012, at 4:03 PM, Steve Piercy - Web Site Builder wrote:
>
>>May I ask how you tracked it down? For example, what clues or symptoms led you
>to discover the root cause? What techniques did you use, e.g.,
>watch network traffic, logs?
>>
>>In Lasso 8.x, one could set SiteAdmin to ignore the -host parameter in
>email_send.
>
>A whole lot of pain.. Initially I thought it was external
>sessions, so we swapped back to internal (on a three week old
>backup of the VM) and the problem went away for about 96 hours.
>
>When it came back, it would occur within 1 minute of lasso
>loading, prior to it service any request. Apache would still
>serve files when Lasso was in a locked up state.
>
>Thinking it was session related, I went hunting through any log
>file I could find.. In Lasso.out.txt I had the following:
>
>Failure in sqlite_session_driver active_tick: Error from SQLite
>database "lasso_session": 19 constraint failed
>[2012-08-14 16:47:41] SMTP: [22] Sending messages to host 127.0.0.1.
>[2012-08-14 16:47:41] SMTP: [22] Done sending messages.
>
>So I thought that it sounded bad, and started digging in
>sqllite.. first I dumped the session db to see if it was a bad
>session, no luck.
>
>Then I started looking at each email db.. and saw there was a
>message in the smtp_queue..
>So I tried to dump it:
>
>delete from smtp_queue;
>Error: database is locked
>
>It was locked because Lasso was going nuts on it.. Stopping
>Lasso allowed me to delete it, and then Lasso has worked ever since.
>
>Looking through the dump (in the previous post) I could see
>that smtp_host was null. This is usually defined by a variable,
>so I created a test file on our development machine that tried
>to do an email_send with a null host, and the same issue occurred.
>
>Some of this was logical, and some of it was luck, but it is
>now fixed in my code, and hopefully can be addressed in a Lasso
>update (I still believe it should either error completely, or
>fail gracefully)
>
>Daniel Parker
>
>
>#############################################################
>This message is sent to you because you are subscribed to
>the mailing list Lasso
>Lasso@lists.lassosoft.com
>To unsubscribe, E-mail to: <Lasso-unsubscribe@lists.lassosoft.com>
>Send administrative queries to <Lasso-request@lists.lassosoft.com>

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- --
Steve Piercy Web Site Builder
Soquel, CA
<web@StevePiercy.com> <http://www.StevePiercy.com/>

#############################################################
This message is sent to you because you are subscribed to
the mailing list Lasso
Lasso@lists.lassosoft.com
To unsubscribe, E-mail to: <Lasso-unsubscribe@lists.lassosoft.com>
Send administrative queries to <Lasso-request@lists.lassosoft.com>