Crunchers Anonymous

Cafe => General Discussion => Topic started by: arkayn on May 15, 2012, 01:15:55 PM

Title: SETI is down again
Post by: arkayn on May 15, 2012, 01:15:55 PM
Looks like the space sciences lab is down once again.
Title: Re: SETI down
Post by: Jason G on May 15, 2012, 04:01:11 PM
some sortof infrastructure upgrades maybe ?
Title: Re: SETI down
Post by: William on May 15, 2012, 05:08:57 PM
http://ucbsystems.org/category/active/

nothing scheduled...

Well, at least caches should be able to cope with an outage now ;D
Title: Re: SETI down
Post by: arkayn on May 16, 2012, 02:08:06 AM
Looks like they are showing it as an unscheduled outage now.
Title: Re: SETI down
Post by: arkayn on May 16, 2012, 02:52:48 AM
QuoteRichard Haselgrove (http://lunatics.kwsn.net/index.php?action=profile;u=164) wrote:
Got some authoritative news.

Quote
It is a short at a substation causing a power outage to the hill area where the Space Sciences lab is located. Traffic signals are out on the hill, all the homes in that area have no power.
That was posted as 'current' about half an hour ago. Since the cricket graph flatlined well over 12 hours ago, they obviously need to do a lot more than just reset a breaker. Even once power is available, there'll be a lot of checking to do as the servers spin up again. Don't expect much SETI action today.
Title: Re: SETI down
Post by: arkayn on May 16, 2012, 09:45:11 AM
QuoteUpdate: Wednesday, May 16, 2012  4:00 pm: Service desk has been informed that the new ETA for restoration of service is Thursday afternoon.

I think this is ridiculous service for PGE, almost 48 hours to fix this in non-critical weather.

They do not even have it listed on their outage map.
http://pge.com/myhome/customerservice/outages/
Title: Re: SETI down
Post by: arkayn on May 17, 2012, 09:42:59 AM
QuoteBrad Deauman (http://www.gpuug.org/content/update-seti-home-outage) wrote:
Here's a quick update on the outage for interested parties:

A 15kV underground power line heading from the campus power substation to the hill that houses the SETI at Home lab shorted out around 2030 on Tuesday night.  PG&E, the local power company, has spent the last two days crawling the line from manhole to manhole looking for the problem.  Apparently while looking for the first short, they managed to cause another short in the line.

PG&E has suggested power should be back working this (Thursday) afternoon however understandably this may not happen on schedule.  Please also keep in mind that it may take some time for the lab to get back into full swing due to this unexpected shutdown.

Thanks go to Dr. Eric Korpela for the above information and for taking the time to fill us in.

Warmly,

Brad Deauman.
Title: Re: SETI down
Post by: arkayn on May 17, 2012, 03:02:44 PM
QuoteUpdate: Thursday, May 17, 2012 11:45 pm: IST Service Desk has been informed that the power has now been restored.

Most likely tomorrow morning they should get the servers up and running.
Title: Re: SETI down
Post by: Jason G on May 17, 2012, 06:03:03 PM
Phew! that's good news  :)  Looks like my ~3.5 day cache was way more than enough this time.  Still only half way through that, so I won;t even have to let my machines hammer the servers during recovery..
Title: Re: SETI down
Post by: arkayn on May 18, 2012, 02:02:20 AM
I am hammering the upload server, but might set NNT for the time being.
Title: Re: SETI down
Post by: arkayn on May 18, 2012, 07:29:50 AM
Getting closer, about 2/3rds of my completed work has been uploaded now.

And done fairly quickly all of a sudden.

Now I just let it report 50 at a time and I will be done.
Title: Re: SETI down
Post by: arkayn on May 29, 2012, 04:31:42 AM
From the front page at SETI@Home...

QuotePlanned Power Outage
Repairs are necessary after the short circuit that caused power failures in several labs here on campus two weeks ago. These repairs require a planned power outage, which starts this afternoon (Tuesday 22:00 UTC). We'll be back on line Thursday morning (roughly Thursday 17:00 UTC). Our websites and data servers will be offline throughout this outage. 29 May 2012 | 17:11:12 UTC
Title: Re: SETI down
Post by: William on June 10, 2012, 11:36:55 PM
Something yellow and fluffy was mentioned earlier today in passing. It took the servers a few hours to get the hint, but they rallied to the cause and went on mass strike.
Amazingly cricket is still showing downloads.
Title: Re: SETI down
Post by: Ex on June 11, 2012, 01:05:55 AM
Anyone have any idea what's up this time?

Perhaps they'll get it up and running when they show up in an hour? Rubber band broke again probably.
Title: Re: SETI down
Post by: arkayn on June 11, 2012, 01:57:44 AM
Quote from: Barry on June 10, 2012, 11:36:55 PM
Something yellow and fluffy was mentioned earlier today in passing. It took the servers a few hours to get the hint, but they rallied to the cause and went on mass strike.
Amazingly cricket is still showing downloads.

I think cricket is off it's rocker as neither of the download servers are reachable.

IST Service Status (http://ucbsystems.org/category/active/) does not list anything on it's site.
Title: Re: SETI down
Post by: Ex on June 11, 2012, 02:06:39 AM
And now the Boinc site is down too...

There was some stuff listed here (http://ucbsystems.org/) as being worked on today, however it should have been done already, and besides it didn't look like anything related to SSL was listed for that work.


So very confusing indeed.
Title: Re: SETI down
Post by: arkayn on June 11, 2012, 02:13:15 AM
Quote from: Ex on June 11, 2012, 02:06:39 AM
And now the Boinc site is down too...

There was some stuff listed here (http://ucbsystems.org/) as being worked on today, however it should have been done already, and besides it didn't look like anything related to SSL was listed for that work.


So very confusing indeed.

I had just checked that site about 30 minutes ago and it was still up. So something is up.
Title: Re: SETI down
Post by: Ex on June 11, 2012, 02:46:35 AM
From Mark S, at Lunatics:

QuoteGot word that it is indeed another power outage.
Duration and cause unknown at present.
Title: Re: SETI down
Post by: arkayn on June 11, 2012, 05:25:03 AM
Again from Mark S.

QuoteApparently, the outage is limited to the building the Seti lab is in, and not directly related to the high voltage lines that shorted previously.
Title: Re: SETI down
Post by: arkayn on June 11, 2012, 08:03:03 AM
And it is back.
Title: Re: SETI down
Post by: William on June 11, 2012, 08:39:43 PM
http://setiathome.berkeley.edu/forum_thread.php?id=68419

As I said - we lost the web/serves before we lost power.
Title: Re: SETI down
Post by: Ex on June 12, 2012, 03:20:37 AM
Probably due to the power browning out... Machines don't like running at half voltage. :-)
Title: Re: SETI down
Post by: arkayn on August 13, 2012, 12:02:23 PM
Quote from: Jeff Cobb


Due to a planned day long power outage, which starts early on Wednesday, all servers will be down from the beginning of Tuesday's normal outage until early on Thursday.

Title: Re: SETI down
Post by: arkayn on October 24, 2012, 01:22:47 AM
QuoteUnable to connect to database - please try again laterError: 2003Can't connect to MySQL server on 'carolyn' (111)

Looks like Carolyn decided to take an extra day off this week.
Title: Re: SETI down
Post by: arkayn on January 03, 2013, 02:27:13 AM
Quote from: Matt LebofskyDue to lab-wide electrical repairs there will be a complete power outage from January 4th to January 6th, during which all our servers will be unreachable. We are also having our server closet air conditioner repaired, and will be completely off line from January 14th to January 15th.
Title: Re: SETI down
Post by: Claggy on January 05, 2013, 06:22:56 AM
Quote from: Matt LebofskyUPDATE: the outage scheduled for the weekend of the 4th-6th has been cancelled so normal operations shall continue. We still plan to have our air conditioner repaired and will be completely off line from January 14th to January 15th during that time.

Claggy
Title: Re: SETI down
Post by: arkayn on January 14, 2013, 04:54:30 AM
Quote from: Matt LebofskyTwo day outage to repair our machine room air conditioner

Projects are down until the work is complete, which should be at some point on Tuesday, January 15.
                                       14 Jan 2013, 17:00:36 UTC
Title: Re: SETI down
Post by: arkayn on February 22, 2013, 12:51:28 AM
Quote from: Matt LebofskyLab-wide Power Outage This Weekend (Feb 22-24)
Once again we are going offline due to electrical repairs in our entire building. We plan to come down Friday afternoon (15:00 local time) and be back up by Sunday afternoon (12:00 local time). All SETI servers will be unreachable during that span.
Title: Re: SETI down
Post by: Mike on February 24, 2013, 03:44:56 AM
Seti is back but very slow.
Title: Re: SETI down
Post by: arkayn on February 24, 2013, 03:50:45 AM
I got everything uploaded, but reporting is a no go.
Title: Re: SETI down
Post by: arkayn on September 30, 2013, 12:29:11 PM
 Active
Unscheduled Outage ââ,¬â€œ Emergency Shutdown Data Center Servers

Outage Type: UNSCHEDULED OUTAGE
Date Submitted: September 30, 2013
Outage Start/End Time: 18:15 ââ,¬â€œ TBD
Groups Impacted: campus
Equipment: Data Center Servers

Description: UPDATE 19:15 ââ,¬â€œ Power is returning to the Data Center but systems remain shutdown, temperature is slowly dropping.  Servers continue to be monitored.

Due to the campus widespread power outage the Data Center servers are experiencing extreme heat impacting central servers residing in the Data Center.  With temperatures rising over 100 degrees , all servers are gracefully being shutdown.

Services Impacted:

Calmail

Bconnected

Applications hosted on VM

All Database Systems

Telecom Gateways impacting Voice Services

BSpace

BCourses

CalPlanning

BAIRS
Title: Re: SETI down
Post by: William on September 30, 2013, 04:21:52 PM
UPDATE 1000pm ââ,¬â€œ The Control-M job scheduling program will be restored at 10:30pm.   Batch jobs for the following systems will likely be delayed impacting CalAnswers, HCM, PPS, BAIRS and BFS.  All other enterprise services continue to be restored and monitored throughout the evening.  Next update will be tomorrow morning.

0900- All Oracle and MySQL Production and Development database servers have been restored.  Most enterprise systems are anticipated to be restored within the next hour.

0800pm- Email Services have been restored.   Temperature is dropping to an acceptable level to begin restoring all services.  Updates will continue as services are restored.

Title: Re: SETI down
Post by: TBar on October 01, 2013, 12:26:49 AM
Seems the incoming UC president's prediction was correct....somewhat. The power grid was 'attacked', and the net went down. Interesting.

(http://i2.wp.com/www.dailycal.org/assets/uploads/2013/09/explosion.FANG_.jpg)

UC Berkeley officials declare emergency after explosion by California Hall (http://www.dailycal.org/2013/09/30/campus-wide-power-outage-disrupts-classes-early-monday-evening/)
Title: Re: SETI down
Post by: William on October 01, 2013, 01:58:26 AM
Webpage is back, project is down for maintenance but left up-/ and downloads running.
Let's hope the colo did an orderly shutdown...
Title: Re: SETI down
Post by: Jason G on October 02, 2013, 01:25:00 PM
OMG Eric's android app exploded, in his office in the sewer  :o
Title: Re: SETI down
Post by: arkayn on October 02, 2013, 01:48:25 PM
Quote from: Jason G on October 02, 2013, 01:25:00 PM
OMG Eric's android app exploded, in his office in the sewer  :o

The current WU I have on my old HTC One V is at 95 hours and only 51% done.

It normally finishes around 48 hours. I am going to see if it will actually finish or crash out eventually.
Title: Re: SETI is down again
Post by: arkayn on October 18, 2014, 05:22:04 AM
Cricket just flatlined and both main and beta are not responding.

(http://fragment1.berkeley.edu/newcricket/mini-graph.cgi?type=png;target=%2Frouter-interfaces%2Finr-211%2Fgigabitethernet6_17;inst=0;dslist=ifInOctets%2CifOutOctets;range=151200;rand=280)
Title: Re: SETI is down again
Post by: Mike on October 18, 2014, 06:15:27 PM
It looks like somebody tried to restart but didn`t work.
Title: Re: SETI is down again
Post by: arkayn on August 17, 2015, 10:47:04 AM
While they are transferring the servers, there is an IRC server set up so anyone who wants to chat, can.

76.101.50.133/6667
#setidown
Title: Re: SETI is down again
Post by: Claggy on August 17, 2015, 01:48:21 PM
Quote from: arkayn on August 17, 2015, 10:47:04 AM
While they are transferring the servers, there is an IRC server set up so anyone who wants to chat, can.

76.101.50.133/6667
#setidown
And the reason was:
Quote
SETI@home Network Outage (August 17-18)
Due to network and DNS upgrades/changes we will be shutting down the projects tonight (August 17th) and we will be offline until the next morning.  18 Aug 2015, 3:46:32 UTC

Claggy
Title: Re: SETI is down again
Post by: arkayn on March 22, 2017, 01:38:24 AM
Right in the middle of a challenge and no work flowing.

Some of the servers came up last night, but the work dried up fast.
Title: Re: SETI is down again
Post by: arkayn on March 25, 2017, 09:53:33 AM
After only SETI being down last night, both SETI and BOINC are down currently.