Author Topic: Site update failed 4/18  (Read 62 times)

0 Members and 1 Guest are viewing this topic.

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 988
  • Last Login:Yesterday at 09:40:27 pm
Site update failed 4/18
« on: April 19, 2018, 07:06:57 am »
I launched a routine site update last night before going to bed and it looks like it caused the DB on the server to become unresponsive.  TM and this forum were both unavailable until I noticed the problem this morning, when I restarted the DB on the server and re-launched the DB update portion of the process.  This was happening occasionally a while back but then we went a long time without a problem.  Now it's happened twice in a few days.  Unfortunately, it's not reproducible so it's hard to track down what's going on.

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 1219
  • Last Login:Today at 12:49:51 am
Re: Site update failed 4/18
« Reply #1 on: April 19, 2018, 11:47:44 am »

Offline dave1693

  • Newbie
  • *
  • Posts: 26
  • Last Login:April 20, 2018, 06:52:05 pm
Re: Site update failed 4/18
« Reply #2 on: April 20, 2018, 06:52:22 pm »
I **hate** HeisenBugs.

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 988
  • Last Login:Yesterday at 09:40:27 pm
Re: Site update failed 4/18
« Reply #3 on: Yesterday at 12:54:35 am »
I'm thinking it's at least not a bug in my update code.  The mysql server hangs right near the start of the ingestion of the new .sql file in a part that hasn't changed in a long time (something like line 4).  My guess is that there's a deadlock in mysql server that gets triggered by some combination of the site update sql starting at precisely the wrong time relative to something else the server is doing (it handles more than just TM stuff on this server).