Author Topic: Site update failed 4/18  (Read 4446 times)

0 Members and 1 Guest are viewing this topic.

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2732
  • Last Login:Today at 02:09:10 pm
Site update failed 4/18
« on: April 19, 2018, 07:06:57 am »
I launched a routine site update last night before going to bed and it looks like it caused the DB on the server to become unresponsive.  TM and this forum were both unavailable until I noticed the problem this morning, when I restarted the DB on the server and re-launched the DB update portion of the process.  This was happening occasionally a while back but then we went a long time without a problem.  Now it's happened twice in a few days.  Unfortunately, it's not reproducible so it's hard to track down what's going on.

Offline yakra

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 4234
  • Last Login:February 13, 2024, 07:19:36 pm
  • I like C++
Re: Site update failed 4/18
« Reply #1 on: April 19, 2018, 11:47:44 am »
Sri Syadasti Syadavaktavya Syadasti Syannasti Syadasti Cavaktavyasca Syadasti Syannasti Syadavatavyasca Syadasti Syannasti Syadavaktavyasca

Offline dave1693

  • Full Member
  • ***
  • Posts: 185
  • Last Login:November 21, 2023, 11:28:54 pm
Re: Site update failed 4/18
« Reply #2 on: April 20, 2018, 06:52:22 pm »
I **hate** HeisenBugs.

Offline Jim

  • TM Collaborator
  • Hero Member
  • *****
  • Posts: 2732
  • Last Login:Today at 02:09:10 pm
Re: Site update failed 4/18
« Reply #3 on: April 21, 2018, 12:54:35 am »
I'm thinking it's at least not a bug in my update code.  The mysql server hangs right near the start of the ingestion of the new .sql file in a part that hasn't changed in a long time (something like line 4).  My guess is that there's a deadlock in mysql server that gets triggered by some combination of the site update sql starting at precisely the wrong time relative to something else the server is doing (it handles more than just TM stuff on this server).