Pre Migration Site Outage

Discussion in 'Announcements, How do I, Feedback' started by Sully, Jul 14, 2017.

  1. Sully

    Sully Administrator
    Staff Member Articles Moderator

    Jul 21, 2015
    2,097
    912
    113
    Software
    30350
    Ratings:
    +1,102 / 8 / -0
    All -

    Over the past few weeks, we have been seeing some very odd behavior on the site. Specifically you as members would see random things stop working like profiles, garage updates, etc stop working. On the backend files were randomly going missing. I thought it was resolved last week. :mad2:

    Last night we had a complete site outage starting around 3:30 - 4:00AM. It appears a large number of files disappeared which required me to do a restore from a backup. I'm in the process of completing the restore. Once that happens, I will see if there is a way to restore the post that were created after the backup.

    I currently do not know root cause. As I move through resolving this I will keep you update.

    Please let me know if you have any questions or find functionality that isn’t working.
     
  2. Sully

    Sully Administrator
    Staff Member Articles Moderator

    Jul 21, 2015
    2,097
    912
    113
    Software
    30350
    Ratings:
    +1,102 / 8 / -0
    Quick Update -

    The hosting provider said the restore is still in process due to the overall size. I'm working with them to try and restore the missing post from the last 48hrs... still working on it.

    Just wanted to keep everyone in the loop. Let me know if you see any issues etc.
     
  3. Crashton

    Crashton Club Coordinator

    Jun 4, 2009
    19,424
    10,031
    113
    Retired old fart
    Hooterville Ohio USA
    Ratings:
    +11,656 / 2 / -0
    Thanks for your hard work Sully. :Thumbsup:
     
  4. MCS02

    MCS02 Moderator
    Staff Member Articles Moderator Supporting Member

    Jul 31, 2009
    11,164
    6,170
    113
    Male
    Ratings:
    +7,876 / 10 / -3
    Yea thanks! I think I need a restore. I wish I had a backup from my 20's :D
     
  5. vetsvette

    vetsvette MINI Alliance Ambassador

    Nov 9, 2013
    2,181
    1,782
    113
    South Central Virginia
    Ratings:
    +1,937 / 0 / -0
    I'd settle for my 30's. I don't think I could survive my 20's again.
     
  6. Metalman

    Metalman Well-Known Member
    Lifetime Supporter

    Sep 29, 2009
    12,713
    7,659
    113
    Ex-Owner (Retired) of a custom metal fab company.
    Columbus, Ohio
    Ratings:
    +7,913 / 1 / -0
    Thank's Sully....

    It's Verking again...:Thumbsup:
     
  7. Metalman

    Metalman Well-Known Member
    Lifetime Supporter

    Sep 29, 2009
    12,713
    7,659
    113
    Ex-Owner (Retired) of a custom metal fab company.
    Columbus, Ohio
    Ratings:
    +7,913 / 1 / -0
    Weird...

    When I posted the thank you...

    Got an error message
     
  8. Metalman

    Metalman Well-Known Member
    Lifetime Supporter

    Sep 29, 2009
    12,713
    7,659
    113
    Ex-Owner (Retired) of a custom metal fab company.
    Columbus, Ohio
    Ratings:
    +7,913 / 1 / -0
    Still acting weird...
     
  9. Sully

    Sully Administrator
    Staff Member Articles Moderator

    Jul 21, 2015
    2,097
    912
    113
    Software
    30350
    Ratings:
    +1,102 / 8 / -0
    Thanks, really hate that it went down. I'm still working on testing different things on the restore.

    Metalman - what type of error did you get?
     
  10. Sully

    Sully Administrator
    Staff Member Articles Moderator

    Jul 21, 2015
    2,097
    912
    113
    Software
    30350
    Ratings:
    +1,102 / 8 / -0
    Ok - looks like I see the error.. error 500. I will have to look into that one. I'm still working on the restore, but that wasn't happening yesterday. I wonder if this is because I posted this in the middle of the restore.

    I'll start a "Test" thread and also check others to see if the "500" errors are happening.
     
  11. Sully

    Sully Administrator
    Staff Member Articles Moderator

    Jul 21, 2015
    2,097
    912
    113
    Software
    30350
    Ratings:
    +1,102 / 8 / -0
    Does this response work, was getting error 500 on the other responses I did on this thread.
     
  12. Sully

    Sully Administrator
    Staff Member Articles Moderator

    Jul 21, 2015
    2,097
    912
    113
    Software
    30350
    Ratings:
    +1,102 / 8 / -0
    OK - this is odd. When you post you get an error 500.. then later the post show up. Working with URLJet on it.
     
  13. Metalman

    Metalman Well-Known Member
    Lifetime Supporter

    Sep 29, 2009
    12,713
    7,659
    113
    Ex-Owner (Retired) of a custom metal fab company.
    Columbus, Ohio
    Ratings:
    +7,913 / 1 / -0
    Same thing you discovered...
     
  14. Sully

    Sully Administrator
    Staff Member Articles Moderator

    Jul 21, 2015
    2,097
    912
    113
    Software
    30350
    Ratings:
    +1,102 / 8 / -0
    All - still having backend DB issues. Working on tracking them.. sorry for the up and down.
     
  15. Sully

    Sully Administrator
    Staff Member Articles Moderator

    Jul 21, 2015
    2,097
    912
    113
    Software
    30350
    Ratings:
    +1,102 / 8 / -0
    Quick update - yesterday I did a partial restoration hoping I would be able to recover post. At this point, it looks like we will need to do a full restore. (this post will get lost in the process).. but for those who are online, wanted you to know its going to start happening soon.. Again I will try to recover the post and once back up.
     
  16. wjosh45

    wjosh45 Member

    Nov 20, 2016
    131
    3
    18
    Ratings:
    +3 / 0 / -0
    Still working on resolving.. I expect a few more issues
     
  17. Sully

    Sully Administrator
    Staff Member Articles Moderator

    Jul 21, 2015
    2,097
    912
    113
    Software
    30350
    Ratings:
    +1,102 / 8 / -0
    OK - so gallery is mostly back up... by that I mean the main page, calendar etc. When you click on the profile, doesn't show anything. Still working on that and root cause. :fingerscrossed:
     
  18. Metalman

    Metalman Well-Known Member
    Lifetime Supporter

    Sep 29, 2009
    12,713
    7,659
    113
    Ex-Owner (Retired) of a custom metal fab company.
    Columbus, Ohio
    Ratings:
    +7,913 / 1 / -0
    Sully...

    When you post to a thread.... You get this error message...

    Screen Shot 2017-07-16 at 6.48.41 AM.png

    But the posting does go through anyway...
     
  19. Sully

    Sully Administrator
    Staff Member Articles Moderator

    Jul 21, 2015
    2,097
    912
    113
    Software
    30350
    Ratings:
    +1,102 / 8 / -0
    Metalman -

    Thank you, I appreciate the update. I had all of it working last night around 2am except the "Profiles." when launched from the gallery page. Somehow in 8 hours we are now seeing this come back. I'm working with URLjet and Nathan to continue to figure out root cause. Ultimately something is deleting the files, I believe its either a corrupt script or a SQL injection (Its my understanding from Nathan, that is what happened a few years ago that created this). Finding that is like finding a needle in a haystack.. but I'm working on this.

    Again - sorry this is happening, but I will get it figured out. Let me know if you are seeing other issues. I'm also walking through different functionality and logs to find issues.
     
  20. Sully

    Sully Administrator
    Staff Member Articles Moderator

    Jul 21, 2015
    2,097
    912
    113
    Software
    30350
    Ratings:
    +1,102 / 8 / -0
    All -

    Quick update and follow up to Metalman's post. When you post you will get an error 500. If you then go back to motoringalliance.com you will see it actually posted. I'm working through that error, but at least it does appear the post does happen. I'm working through this and hope to have it resolved today if possible.
     

Share This Page