MPC Status Page
This page describes enhancements to and problems that have occurred with the
MPC's webpages
and scripts and the fixes that have been made. Resolved problems will
remain in the Outstanding Problems sections for a brief period after
resolution before being moved. This page will also try and give advance
notice of the unavailability of these services due to network maintenance.
Known problems with contacting certain e-mail addresses are listed
elsewhere.
Unless otherwise indicated, all times on this page are UTC times.
Prior to 2010 Nov. 7, the times shown are local times
(either Eastern Standard Time [UTC-5h] or Eastern Daylight time [UTC-4h], as
appropriate).
Use the feedback form to report
new problems or
to comment on this page.
Status Messages and Outstanding Problems
- Unexpected Power Interruption May 16
2013 May 16: 16:00. We had one hour's notice that power to the building where
part of our computer system resides would be turned off to allow engineering
work. We were able to stop the automated processes and dismount the
shadow sets, but were unable to stop the machines restarting when power
was restored. This uncontrolled power up has caused major problems to
the cluster. One machine is off line (service has been called) and one
external disk box is not sharing its disks (one member of each of four
three-member shadow sets). No data has been lost, but
until the affected machine has been repaired, certain internal tasks
may be delayed.
- One feature that we cannot easily replicate while the off-line machine
is unavailable is complete datasets necessary for the various MPChecker
facilities. Our internal checking routines are not affected by this
problem.
- There is a workaround in place for the MPChecker services. Note that
full functionality is not yet back in place as the service is missing
the nearby-epoch data for NEOs and the historical data for all objects.
- Network Interruption May 14
2013 May 14: 16:00. We have been reminded that the main network router at
60 Garden Street will be upgraded during the time period 21:30 UTC on May 14 to 01:30 UTC on
May 15. MPC webservices should remain accessible during this network
outage, but updating of pages will not be possible due to internal
network disruption.
- Potential Service Interruptions May 18
2013 May 14: 13:25. We have been informed that Computation Facility services will be
unavailable between 13:00 and 18:00 UTC on Saturday, May 18. CF webservices
and e-mail service will be unavailable during this period. The MPC webservices
(and the updating of data used by the services) should be unaffected by this
downtime.
- SKYCOV Restored
2013 Apr. 8: 16:26. The SKYCOV service has been restored to full functionality and
sky coverage data is again available up to the current date.
- SKYCOV and DELCODE Removals
2013 Apr. 6: 01:52. We are now looking at repairing the SKYCOV service. This will also
fix the problem of the automatic removal of DELCODE requests. In the meantime, the
script that validates DELCODE requests for removal will forward the request to a
staff member for removal.
- Apr. 7: 13:45. Automated removal of DELCODE requests was reactivated last night.
Please report any problems with removals.
- Apr. 7: 15:45. The SKYCOV service is almost restored. Batches of sky coverage
are being copied to a temporary directory on the app server. We just need to move
those copied files to their final locations and rebuild the catalogue of available
files. These tasks will be done tomorrow.
- File Protection Problems on Web Server
2013 Mar. 27: 12:15. Following the updating over the past two nights of various files on the
web server machine, it appears that the transfer process has suddenly started disabling
world/other read permission on two of the many files that it copies. This is annoying
as it prevents the MPES from running. This problem has occurred in the past, for
different files over varying periods of time. We are investigating the best way to
locate problem files and reset the permissions if found to be wonky.
- 12:45. The potential source of the problem has been identified and a fix has been
implemented. As to why only two of four transferred files (all unpacked into the same
directory) were affected by this problem, we have no idea!
- AUTOACK Problem
2013 Mar. 25: 20:32. The machine that sends out ACKs and processes incoming e-mails
is having hardware issues. We are investigating and attempting to move processes
to another machine.
- Mar. 26: 16:31. A rebuild of the process was required. A functioning version
of AUTOACK is now running on another machine. Some recent observations batches have been ACK'ed
by this new routine running in test mode. Other observation batches that arrived after Mar. 25
17:16:21 UTC and before March 26, which were processed manually during the outage, will be
forwarded to AUTOACK. Observers may receive two (or more) ACKs for messages. A semblence
of normalcy has been restored, further work will be necessary to add functionality to the
new AUTOACK process.
- Mar. 26: 14:10. The SKYCOV service has also been affected. A partial fix to this
problem is forthcoming (accepting of incoming e-mails), but a full fix will have to wait
until after the March Minor Planet Circulars are prepared.
- Mar. 26: 20:14. Extensive testing of the new routine showed up some problems, which have
all been fixed. Automated ACKs have resumed.
- E-mail Submission of Observations Via mpc@cfa Shut Off
2013 Mar. 4: 13:32. As noted on MPEC 2013-A21,
the e-mail address mpc@cfa can no longer be used for submission of observations.
E-mail submission of observations must be via obs@cfa. Any future submissions
of observations to mpc@cfa will not be acknowledged. The e-mail address
mpc@cfa will continue as a general contact address for the MPC.
- No Mid-Month MPS Batch
2013 Feb. 8: 23.55. As a direct result of this weekend's major snow storm in
New England and the consequent major risk of extended power outages, there
will be no mid-month MPS batch issued this weekend.
- Problem With MPES
2013 Jan. 30: 15:14. It appears that there is a problem with one (or more)
of the data files used by the MPES. We are investigating.
- 16:14. The problem with the MPES data files has been located and
fixed.
- Intermittent Network Problems
2013 Jan. 9: 22:40. We have received a report from our CF that Harvard
is investigating intermittent network connectivity problems. We have
already received a report of a script being temporarily inaccessible.
- Extended Power Outage at CDP
2013 Jan. 3: 15:01. There was a four-hour power outage at CDP between
07:30 and 11:20 this morning. The website and scripts should again
be visible. Internal operations are being restarted as necessary.
- 21:10. We have been informed that the power issue this morning
fried one of the main electrical power fuses in the computer room.
It will be replaced at 22:00 on Monday, January 14. This will
require shutting off the power. The downtime is expected to be
under 30 minutes, but one hour is the announced timeframe. Website
and scripts will be off-line from around 21:30 until power is restored.
- December 28 MPCAT-OBS Comet File Replaced
2012 Dec. 30: 21:50. Following a report from a user that the MPCAT-OBS file
of comet observations was the same as that from the previous month, the
file has been replaced.
- December 28 DOU MPEC
2012 Dec. 28: 06:00. Tonight's DOU MPEC will not be issued, to avoid
any possible conflict between its processing and the processing associated
with the filing of the Dec. 28 MPC batch which is still running.
- Malformed MPCAT-OBS Files (December 19)
2012 Dec. 19: 23:45. A number of users have informed us that the mid-month
complete observation files associated with the 2012 Dec. 16 batch of
MPS are corrupted. We have tracked down the reason for this
corruption and are regnerating the files.
- Dec. 20: 03:53. Corrected files have been put on-line.
- SMTP E-mail Interruption (December 18)
2012 Dec. 17: 16:45. We have been informed that the CF's SMTP server
will be off-line for approximately 60 minutes beginning at 20:00 UTC
on Tuesday, December 18, to allow replacement of a faulty CPU. Attempts to
send e-mail to us during this period will fail, but the mail should
queue up on the sending computer for a subsequent resend attempt.
- Missing Objects in MPChecker
2012 Nov. 1: 16:00. The previous problem of missing objects in the MPChecker
service has resurfaced. We are investigating.
- Fixes for a Number of Problems
2012 Oct. 23: 15:00. A number of fixes for minor web-based problems
have been made.
- The lists of Trojan objects are working again.
- Recently-discovered comets are accessible in both the MPES
and MPChecker.
- The complete file of skycov pointing data has been moved from
the CF anon-ftp site to our own web server. The updating of this
file is now on a monthly schedule.
- Last Night's DOU MPEC
2012 Oct. 17: 14:00. Last night's DOU MPEC has been abandoned
due to an internal problem.
- Power and Network Problems
2012 Oct. 14: 10:00. There are problems with the MPC machines over at 60
Garden Street. We will investigate as soon as possible.
- Oct. 15: 03:00. A UPS has failed. This caused one machine to go
off-line and three others to reboot. In addition, the main switch
controlling the small intra-net is having problems: it seems to be
resetting itself every 10 seconds, which is preventing the rebooting
machines from coming up cleanly.
- 04:00. The shadow sets are rebuilding, we'll see what progress has
been made in the morning.
- 09:30. No progress on the shadow sets. The cluster is being
rebooted. Ensuring that nothing is left from incomplete runs of automated
processes will take a while.
- 15:00. The final bits are being restarted.
- MPEC 2012-O37 Resend
2012 July 31: 19:37. MPEC 2012-O37 has just been remailed. The initial
mailing failed: the VMS side handed the necessary files over to mpcapp1;
mpcapp1 handed off the relevant files to the mail system; the mail system
handed off the files to the Harvard mail server; then nothing. We will try
to find out what went wrong on the Harvard side.
- Temporary Move of Webserver
2012 June 26: 19:56. The MPC webserver mpcweb1 is having issues. While we
investigate (and fix!) these problems, the webserver has been moved to
our database machine, mpcdb1 (131.142.195.57). A temporary change has
been made in the DNS records, but this change may take time to propagate.
Users should continue to access URLs at www.minorplanetcenter.net as
usual.
- 20:18. We now see the new IP addresses on our home systems.
- June 27: 03:07. We are moving the hostname www.minorplanetcenter.net
back to mpcweb1.
- Customize Page
2012 June 15: 03:41. A user reported a problem with the Customize page script.
Incomplete pages would be returned. The problem has been identified and
fixed.
- Time Issue on Scully
2012 June 6: 16:18. The system time on scully seems to be misconfigured.
Displaying the date and time shows a UTC time, but the timezone is
indicated as EDT. This is confusing the cgi scripts.
17:45. The system timezone has been set to UTC. Cgi scripts should
behave normally.
- Problem With (Some?) Odd-Epoch Elements in MPES
2012 May 18: 15:30. As a result of a MPC member noticing a message detailing
a problem with an MPES ephemeris on a mailing list, it has been determined
that some comet elements may have been incorrectly integrated to a
current epoch. The use of a current epoch in the MPES is intended to
allow observers to generate their own unperturbed ephemerides, with
reasonable results. The problem seems to be related to the recent
procedure (running under Linux) introduced for producing odd-epoch
elements. As a test, we are regenerating the odd-epoch comet elements
using the old method (running under OpenVMS).
- 17:47. The offending data file has been replaced. The correct
elements and ephemeris are now returned in the MPES. The orbital elements
for software packages have been regenerated. We are trying to reproduce
the error on the Linux side that caused this problem, but have not yet
succeeded.
- Webserver Rebooted
2012 Apr. 30: 18:15. The MPC webserver machine has been rebooted, due to
another critical memory shortage, presumably caused by a memory leak. The
condition was flagged by a monitoring tool, but we are investigating
the causes for the leak.
- May 3: 14:30. We have identified a potential cause for the reboots.
It seems that certain search engines are ignoring the robots.txt file
and are pounding the DB with thousands of requests that return large
amounts of data. The IP addresses (and sub-domains) of these problem
search engines have been blocked. Further blocks may be put in place
as needs arise.
- Webserver Rebooted
2012 Apr. 23: 14:37. The MPC webserver machine has been rebooted, due to
a critical memory shortage, presumably caused a memory leak. The
condition was flagged by a monitoring tool, but we are investigating
the causes for the leak.
- Network Problems at CDP?
2012 Apr. 18: 02:40. We are receiving reports that MPC webpages and
services are unavailable. The public-facing machines are up and
running. We are not yet sure what the problem is, but are
investigating.
- 04:10. The webserver machine has been rebooted. Pages are
being served as normal.
- Sending of Designations II
2012 Apr. 10: 01:30. The datafiles necessary to allow the resumption of
the AUTODES service have been recovered from the failed system. The
automatic sending of designations has resumed.
- Recent MPEC non-mailing
2012 Apr. 1: 15:00. Recent MPECs do not seem to have been mailed out
via mpcapp1. The affected circulars are numbers 2012-F107 to 2012-F114,
and 2012-G01 to 2012-G06. The necessary files for each mailing have been
copied from the VMS side to the Linux side (confirmed by examination of
the VMS log files) and the circulars are up on the website, so the VMS
side is working normally. The Linux system thinks that they've been
e-mailed, since the files associated with each mailing that get deleted
after the mailing have been deleted. We are investigating. The DOU
MPECs are unaffected by this problem, as they continue to be
mailed out via the old route. Use of the new route for the DOU
MPECs, which was planned for the next few days, will be delayed.
- 16:15. MPECs 2012-G08 and -G09 have been e-mailed without
problem. No modifications were made to the mailing process on the Linux
systems. The missing MPECs will be re-mailed later today.
- Apr. 2: 03:06. All the missing MPECs have been remailed.
- Apr. 11: 15:05. The DOU MPEC is now being issued via the new
route.
- Multiple e-mail copies of MPEC 2012-F75
2012 Mar. 23: 19:05. The first MPEC (issue 2012-F75) has just been
issued via the new mpec@mpcapp1 route. The script that issued this
circular was run several times, since it appeared that nothing was
happening and that the script had crashed. This was apparently not
the case and multiple copies of this circular were sent out. There should
be only one e-mailed copy of MPEC 2012-F76.
- Sending of Designations
2012 Mar. 16: 16:45. The failure of a machine has caused the automated
designation sender to cease functioning. It may take some time for
this service to resume operation, as we have to recover data from
a disk.
- Network Outage Planned (Thursday Feb. 9)
2012 Feb. 8: 15:40. We have just been informed that there will be a
two-hour network outage at CDP (where the MPC is now based) from 09:00
UTC (04:00 EST) on Thursday, February 9. This outage is to allow an
upgrade of the network router that connects CDP to the CfA site.
There will be no network connectivity to/from CDP during this window.
- Missing Orbits in MPES
2012 Feb. 8: 14:30. We have received a couple of reports of problems
accessing orbits in the MPES. The problem has been traced to an
incorrect build of a monthly data file. The affected file is being
rebuilt.
- 19:00. The problem has been fixed.
- Problem accessing www.minorplanetcenter.net
2012 Jan. 28: 01:30. A number of people have reported getting no response
from www.minorplanetcenter.net since about 00:20. We see the same
behavior. The machine is up and accessible. It appears to be a routing
problem within the Harvard network. We are investigating.
- 02:30. The problem seems to have resolved itself.
- January 26 DOU MPEC
2012 Jan. 26: 13:30. The January 26 DOU MPEC was abandoned due to a
failure early on in the production procedure.
- January 12 DOU MPEC
2012 Jan. 12: 16:00. The January 12 DOU MPEC was abandoned due to a
failure early on in the production procedure.
- CfA Network Problems?
2012 Jan. 11: 03:20. We are experiencing network problems in accessing
our work accounts from home. We are receiving reports that outside
users are experiencing issues accessing the MPC web site. We are checking
whether there are network problems at CfA with our Computation Facility.
Access was restored after some hours. No explanation for the problems
has been forthcoming.
- December 31 DOU MPEC
2011 Dec. 31: 15:54. The December 31 DOU MPEC was abandoned due to a
system failure early on in the production procedure.
- Network Outage Planned (Monday Dec. 19)
2011 Dec. 15: 20:50. We have just been informed that there will be a
four-hour network outage at CDP (where the MPC is now based) from 23:00
UTC (18:00 EST) on Monday, December 19. This outage is to allow the
building landlord to re-route network cables into the building. The
outage will affect access to the MPC website and web scripts. The
AUTOACK process will also be disabled during this outage.
- New Object Ephemeris Generator
2011 Nov. 29: 23:05. A corrected executable has been put in place for the
New Object Ephemeris Generator script. The long-standing problem of
ignoring a user's motion options has been fixed.
- Problem with MPC Database
2011 Nov. 27: 15:30. There is a problem with the MPC DB search feature.
This is related to additional information that is being incorporated
into the DB. The relevant person has been alerted.
- 16:35. Normal functioning of the MPC Database has been
restored.
- Problem with Lightcurve Database
2011 Nov. 26: 03:00. Following a report from a user, it is apparent that
there is a problem with the software controling the Lightcurve Database.
The relevant person has been alerted.
- 03:55. Normal functioning of the Lightcurve Database has been
restored.
- Forthcoming E-Mail Outage (Friday Nov. 25)
2011 Nov. 24: 03:30. We have just been informed that the Computation
Facility will be installing OS patches between 11:05 UTC
and 11:35 UTC on Friday, Nov. 25. During this time period
e-mail messages will not be delivered (either inbound or
outbound). Incoming e-mail should be redelivered
automatically once the work is completed.
- Minor Planet Ephemeris Service not functional
2011 Nov. 14: 15:45. A number of users have reported that the MPES has
been non-functional since 06:30. We are investigating.
- 15:57. An examination of the script-server error logs shows
that the first instance of this problem occurred at 06:17:53 UTC.
- 16:23. Some garbage was inadvertantly added to one of the
monthly work files. This garbage has been removed from both the
master monthly work files and from the versions used in the MPES.
The MPES should now return ephemerides as normal.
- Incorrect Dates of Discovery and Sites for Numbered Minor Planets
2011 Nov. 11: 20:26. Following a report from a user about apparent incorrect
discovery circumstances for newly-numbered minor planets, we are checking
the details for 107 suspect objects.
- Nov. 11: 22:35. A number of published discovery details will require
modification. We will make the changes directly in the master
discovery files and document those changes in the next batch of
Minor Planet Circulars. We are also investigating why the problem
occurred.
- Nov. 12: 17:35. The reason for the problem has now been determined
and the documentation for the preparation procedure has been modified
to better handle the issue. We can now investigate what changes to
the discovery details need to be made.
- Nov. 24: 03:25. The discovery details for 77 numbered minor planets
have been corrected in the on-line list. Corrected details will appear
in the December batch of Minor Planet Circulars.
- November 9 DOU MPEC
2011 Nov. 9: 11:00. The November 9 DOU MPEC (2011-V33) is being mailed out.
It is large (~5.6 MB) and this is causing many mail servers to refuse
delivery. Given the size of the circular, it may take some time for
the mailing process to complete. MPEC 2011-V33 is the largest such
circular ever issued and is almost twice the size of the previous
"record holder", MPEC 2001-W71.
- November 8 DOU MPEC
2011 Nov. 8: 13:00. The November 8 DOU MPEC was abandoned following a
problem with production caused by the huge number of orbits.
- Missing Objects in MPChecker
2011 Oct. 24: 18:00. A user reported that entering an observation
of a newly-designated minor planet into the MPChecker did not produce
a match to that object. The problem was traced to the build procedure
of the data files for the webserver. This problem has existed since
Oct. 11. The problem has been fixed and updated datafiles have been copied
over to the webserver.
- Väisälä Orbits in MPES
2011 Oct. 14: 15:30. We have received a couple of reports of problems
with Väisälä orbits in the MPES. The first and last
dates are shown as -4712 Jan. 1 and the absolute magnitude is zero.
These quantities appear to have been trashed on the orbit records for
some Väisälä objects on the webserver. Not all objects
are affected. We are investigating.
- 15:50. We have identified the problem, an issue with building the
secondary datafiles for the webserver. The fix will be visible externally
after the next DOU MPEC. In the meantime, the ephemeris positions
of the affected objects are correct.
- Publication of 2011 Oct. 12 MPCs
2011 Oct. 13: 15:30. A couple of problems have surfaced with the
on-line publication of the 2011 Oct. 12 MPCs. The protection
on the complete file of observations of numbered minor planets was
incorrect, so that outside users could not access it. In addition,
the file itself was truncated (being only about half the correct
length). The first problem may have been a consequence of the second.
Both problems have been fixed. Unfortunately, there are no log files available to check,
since the file was prepared on a Linux system and transferred to
another Linux system. So we have no idea what went wrong and we cannot
be sure that this problem will not reoccur in the future.
- Forthcoming E-Mail Outage (Wednesday Oct. 19)
2011 Oct. 12: 16:03. We have just been informed that the Computation
Facility will be performing disk upgrades between 21:30 UTC
and 23:00 UTC. It is unclear if this maintenance will affect
SMTP e-mail delivery. If it does, messages will not be delivered
during the above-mentioned period, but should be redelivered
automatically once the work is completed.
- Repeated Observations in DataBase
2011 Sept. 15: 19:05. A user reported finding duplicate observations
when extracting observations via the DataBase interface. Strictly
speaking, these are not duplicate observations as the records
differ in the reference. In each case, one of the observations
appeared on an MPEC, while the other appeared on a
MPS. Clearly, the MPEC-published observations
are not being replaced by the MPS-published observations.
We are examining this problem. The duplicate observations are not
present in our master files (copies of which are available via the
MPCAT-OBS service). The duplicate observations in the DB will
be removed shortly. Another problem that we will fix is related
to the insertion of some observations that do not seem to have
been inserted correctly.
- Forthcoming Network Outage (Sunday Sept. 18)
2011 Sept. 15: 18:25. We have just been informed that the Computation
Facility will be performing network maintenance between 11:00 and
12:00 UTC on Sept. 18. Connections to our web services will be unavilable
for up to 30 minutes in this window.
- Sept. 18: 13:50. Automated processes and execution queues were
shutdown beginning
at 10:00 to try and prevent problems with remote resources being
inaccessible. All execution queues have been restarted and the
automated processes are restarting.
- Planned Outage in AUTOACK
2011 Sept. 15: 14:30. There will be a brief outage in the AUTOACK service
between 16:00 and 17:00 UTC. This is to allow a NIC to be replaced.
AUTOACK is the automated process that accepts incoming e-mail and
acknowledges observation batches. Assuming no problems with removing the
old NIC, downtime should be about 10 minutes. No special action is
required by submitters of observations.
- 17:05. This work has been postponed due to the need for further
examination of network errors.
- Problem with MPES and Comet Fragments
2011 Aug. 20: 18:30. A user reported that requesting "213P" from the MPES
resulted in the elements/ephemeris for "213P-B" being returned. This
is now fixed.
- "Customize This Page" Problem
2011 Aug. 15: 22:20. A user reported errors when attempting to use the
Customize This Page script. A simple recopy of the executable fixed the
problem.
- July 26 DOU MPEC
2011 July 26: 14:00. The July 26 DOU MPEC was abandoned following the
unplanned reboot of a cluster machine overnight.
- Problem With Designation Linking in MPC Database
2011 July 22: 22:22. The identification/double-designation table in the MPC
Database is corrupted and will have to be rebuilt. This rebuild will
occur in the near future and will mean that DB queries using non-principal
designations may not return the expected results until such time as
the table is rebuilt completely. Some small insertions into the new
table will be used to check that all is well before the main bulk of the
identiications are inserted. We apologize for the inconvenience.
- July 23: 16.00. The test insertions into the new table showed the
same problem. A close inspection of the relevant script revealed the
reason for the mis-linkings. A fix is in the works, then the table
will be rebuilt anew.
- July 23: 18.20. The identification/double-designation table has
been rebuilt. Tests show the problem cases identified earlier are now
handled correctly. Please report any anomalies.
- Problem With Script Server
2011 July 18: 15:00. A number of users have reported problems accessing
the webscripts. We are investigating.
- 16:00. The problem was traced to a malformed data file. It
was replaced and a number of zombie jobs were deleted.
- Problem With Certain Objects in MPES
2011 July 16: 15:30. A user has reported problems accessing ephemerides for
unnumbered minor planets in the MPES. The problem is due to the non-instantaneous
nature of the updating procedure associated with the filing of material from a
MPC batch. The last bits of the filing are being performed now.
- 18:20. Generation and copying of data files is complete.
- Protection on Complete File of Numbered Minor Planet Observations
2011 June 20: 15:50. A user reported an Error 403: Forbidden error when trying to
access the complete file of numbered minor planet observations. It seems that
the copy procedure that placed the files in that directory set the protection
for that one file (alone!) to no access for anyone other than the owner. All
the other files transferred at the same time had correct protections. The
protection has been reset to read access for all.
- Problem With Certain Objects in MPES
2011 June 16: 18:15. A number of users reported problems accessing ephemerides
for objects with Väisälä elements. A request for
2011 LN3, produced output for 2011 KD41. The problem, now fixed, was
traced to a failure to unpack a zip file containing updated index files
for the orbital elements. This was a transient problem. The sftp system
does not report a useful error message to the OS. A workaround for this
glaring limitation has been implemented.
- Shutdown of www.minorplanetcenter.org
2011 June 7: 16:50. The www.minorplanetcenter.org site has been shutdown.
Traffic to that site will now be redirected to the .net site. It may
take some time for the redirect to make its way around the name servers.
- Network Reconfiguration
2011 June 4: 13:00. There may be temporary outages in the web services over
the next hour or so while we do some reconfiguration of the network
connections for the public-facing web machines.
- MPES Problem
2011 May 11: 15:00. We have received a number of reports that the MPES
is returning incorrect orbits for objects designated in 2011. We
are investigating.
- 15:55. The problem was traced to a .zip file copied to the
script server that was not unpacked. Normal service should have been
resumed.
- Ubasti Retired
2011 May 2: 18:00. We have been informed that access to the http port on
the old webserver (Ubasti) has been disabled by the Computation Facility.
Please use the corresponding pages on the .net site. With one exception, all
services have been ported to the new server. The one exception is the MPCOBS
service: users should extract the observations they require from the
MPC Database interface.
- New Object Ephemeris Generator
2011 Apr. 30: 22:15. We have received a couple of reports of problems with the
new New Object Ephemeris Generator script. These include: misreading
the date if no date is entered and hourly output of ephemerides is
selected (fixed); incorrect motions (the motions are always displayed
in "/hour, regardless of the selection); and incorrect local circumstances.
Fixes are being worked on.
- Incorrect Dates of Discovery and Sites for Numbered Minor Planets
2011 Apr. 21: 21:25. Following a report from a user about apparent incorrect
dates of discovery for four recently-numbered minor planets, we
performed a check to uncover five additional cases. The objects
affected are (264210), (266779), (266959), (266966), (266976),
(275353), (275405), (275426) and (275478). The discovery site (and
the discoverer in one case) for (264235), (266971), (273868)
and (275212) require correction. All these corrections will
be documented fully in the next MPC batch, but the changes
have been made in our internal files and in the files that are
accessible through the website.
- E-mailed Problems with MPECs
2011 Apr. 21: 20:00. A number of users have reported non-receipt of recent
MPECs. We don't see anything obviously wrong at this end, so
the problem may be something transient with the CfA network. We will
monitor the situation.
- Problem with Minor Planet Designation Converter
2011 Apr. 16: 14:00. A user has reported that the Minor Planet Designation
Converter is returning results for a different object than that
requested. The problem is due to a mismatch between the index file and
data file that contains the information. This problem will be examined
after the filing for the April MPCs is completed, so that fresh
copies of the relevant files are available for examination.
- SMTP E-mail Interruption (March 31)
2011 Mar. 30: 13:45. We have been informed that the CF's SMTP server
will be off-line for approximately 15 minutes beginning at 10:05 UTC
on Thursday, March 31, to allow OS patches to be installed. Attempts to
send e-mail to us during this period will fail, but the mail should
queue up on the sending computer for a subsequent resend attempt.
- .org vs .net
2011 Mar. 28: 02:32. We have put redirects from many of the /iau HTML pages
on the .org site to funnel users to the .net site. This is a preliminary
step to the shutting down of the physial .org site.
- Old Epoch Elements for Numbered Periodic Comets in Comet Element Files
2011 Mar. 24: 13:20. An observer has pointed out a problem with the comet
element files accessible in the Ephemerides and Orbital Elements
service. We have determined that the problem is with the epoch of
some of the orbits. We are investigating and will regenerate the
files when the solution is found.
- 14:50. Corrected files have been put in place.
- Retirement of Ubasti Web Server (April 18)
2011 Mar. 23: 15.25. The http server on the old web server machine (the
old destination for the alias 'scully') will be shutoff sometime on
Monday, 18 April. This date is not open to change as it is dependent on
changes in the CfA's internal network that will be implemented at the same
time. This unchangeable (by us!) date has now been changed twice, once
to allow more testing of the internal-network changes and once because
of the uncertainty of whether the U.S. Government would be shut down.
Most cgi scripts are now running on the new script server.
The status of porting the cgi scripts is detailed
here.
- Date of Last Update to NEOCP Not Updating on Main Page
2011 Mar. 16: 14:00. It has been brought to our attention that the
date of last update to the NEOCP that is present on the main MPC
observers page (/iau/mpc.html) has not been updated since Feb. 28.54 UT.
We are investigating.
- 20:40. The problem has been located and, we believe, fixed.
- AUTOACK Problem Overnight
2011 Mar. 12: 18:00. Yesterday evening, the AUTOACK process started failing.
No ACKs were being sent out. No observer brought this to our attention
until early this morning. We have located the problem, fixed it and
have resubmitted the affected batches to AUTOACK. If you use the AC2
keyword in your headers, you should receive the ACK. If you don't use
the AC2 keyword, you will not.
- .org vs .net
2011 Feb. 22: 22:00. Over the next month we will begin to cease updating
certain pages on the .org site, in anticipation of removing the
physical site in the near future. The .org site was set up as
an emergency host following the events of last February. The hosting
site has a number of problems, principally the throttling of transfer
rates for large files, that cause us particular grief in trying to
maintain both the .net and .org sites. The first pages to be
updated only on the .net site will be the ECS pages associated with
mid-month MPS and monthly MPC batches. These .org pages will
be modified to redirect automatically to the .net site. When the
physical .org site is removed, it will be set up to redirect to .net.
- Problem With Certain Objects in MPES
2011 Jan. 31: 15:20. There seems to be a problem with the return of
ephemerides of certain recently-designated objects in the MPES.
A request for an ephemeris just sits at the "waiting for a response"
message. Other objects are unaffected and problems (based on
personal experience) appear to occur only with certain hosts. At this
stage, we do not understand why the problem is occurring.
- Feb. 1: 01:00. We discovered that the problem was due to
incorrect protections on the variant orbit files. The immediate problem
was fixed and the nightly update procedure was modified to set the
correct protections.
- The MPC Web Site
2011 Jan. 19: 15:30. The main MPC web site is now
http://www.minorplanetcenter.net/. This link will take you to the
new interface, with access to new features such as the MPC Database.
For those who are used to the earlier incarnation of the site, the
former page layout is accessible
at http://www.minorplanetcenter.net/iau/mpc.html. The ".org" site
will, at some point in the near future, redirect users to the .net
site.
- Move over to new script server II
2011 Jan. 5: 16:00. Most of the reported problems with the changeover
have been fixed. Variant orbits are currently being copied, to be
followed by residual blocks, for use in the MPES. The problem with
incorrect elements being returned for certain unnumbered minor planet
designations is believed to be fixed.
- Move over to new script server
2011 Jan. 4: 17:15. We have made a request to the CfA's networking group
to shift the machine name "scully" (which is an alias) to the new
script server machine. Some scripts will continue to run on the
old server machine for sometime longer. It is to be expected that quite a number of
links will need fixing. Please use the feedback
forms to report problems. We had hoped that the change over would
be completely transparent to outside users, but the change in the webserver
from Purveyor to Apache makes this impossible. Some
notes on the changes are available and contains details on how
to modify any user-written scripts that call the MPC scripts directly.
- Certain numbered orbits in MPES
2011 Jan. 3: 15:23. Due to a repeated set of elements for a numbered minor
planet, the MPES was returning the incorrect orbit for requests for
numbered minor planets with numbers above (217439). That incorrect
misbehavior has been fixed. A problem remains in that the name or
principal desigantion associated with the numbered object is being
displayed incorrectly.
- Power and Network Outages Dec. 11 and 13
2010 Dec. 10: The power to CDP will be disconnected from 07:00 EST to 15:00
EST on Saturday, December 11, in order to connect a new electrical
service to the Park. In addition, CDP will suffer a network outage
from 18:00 EST to 21:00 EST on Monday, December 13, in order to
relocate the fiber that provides Internet access into CDP. Neither of
these events will (should!) affact core MPC operations, but beta
testers of the new web server will be unable to access the new setup
during these outages.
- Last night's DOU MPEC
2010 Dec. 10: 13:00. Last night's DOU MPEC has been abandoned as a
result of a protection issue in copying some important update files
over to the Linux systems.
- Incorrect H values
2010 Dec. 8: 23:50. Following a report from a user, a problem with the
H values for 143 orbits of numbered minor planets published in the
November MPCs has been identified and been fixed. The H values
on the affected orbits have been
fixed in our master files and in the files used by the MPES; the
values used by the MPChecker programs will be fixed
by the next DOU MPEC. The corrected orbits will also be
republished on the next DOU MPEC.
- November MPCs
2010 Nov. 20: 14:20. The preparation of the November batch of Minor
Planet Circulars will be delayed by a week due to recent events.
- MPChecker
2010 Nov. 14: 22:05. In order to get the newly-hosted MPChecker script
working, we need to introduce a change in an internal data file that
will break the existing MPChecker script. As soon as the new format
data files are on-line, we will fix the existing script.
- 23:05. In a testament to the OS design, the existing MPChecker script
works with (what we thought would be an imcompatible change in the
format of) the new-format data files used by that service.
- Erroneous designation messages
2010 Nov. 9: 12:40. Over the past day or so, some automated process has
been appending unnecessary designation lines to the files that are
used by AUTODES to send designations back to observers. We are
trying to locate which automated process is misbehaving.
- PHAs not being flagged in MPES
2010 Nov. 9: 12:00. Following a query from a user of the MPES, it was
determined that the MPES was not indicating that recent discoveries
were PHAs. The problem was traced to a change in an internal
procedure. This problem is believed to be fixed.
- Change in time-keeping
2010 Nov. 7: 18:00. Following an overnight problem with the automated
changeover of system clocks from EDT to EST, the decision was made to
set all system clocks to UTC. The problem caused five machines to
remain set to EDT and this caused the despatch of the many automated
batch jobs to occur only once per hour, rather than once every minute
or two. All automated batch jobs are now running on their normal
schedule.
- No MPC Batch in October
2010 Oct. 19: 17:50. There will be no MPC batch in October. The
next mid-month MPS batch will be issued around October 31.
- Apparent Unavailability of Scully
2010 Oct. 12: 11:35. Over the past few days we have received a number of
reports of the apparent unavailability of Scully. In every case, a
check of the server logs shows that Scully was serving documents
throughout the reported problem period. It seems that every report
was made at a time of very heavy server load. We are currently
setting up a new (much more powerful) web server machine that we
hope to have on-line shortly.
- 20:10. A check of yesterday's server log shows that 16642 calls
to the cgi scripts from 4580 different IP addresses.
- Complex-wide Power Outage Saturday, September 18
2010 Sept. 14: The power to most of the buildings at 60 Garden Street will be
shut off from 07:00 to 19:00 EDT on Saturday, September 18, in order to
perform a power systems upgrade. The Computation Facility will begin
shutting down its systems at 05:00. While the power outage does not
affect any of the MPC's computer systems (since they are in locations
that will be unaffected by the power outage), the loss of power
elsewhere will cut all network traffic to/from CfA. We will not be
able to send/receive e-mail during the outage: any messages sent to us
should be queued automatically by the sending system and resent once
network connectivity is restored. All CF webservices will be unavailable.
Access to MPC cgi scripts will not possible due to the network outage.
A similar power outage is planned for Saturday, October 2.
- MPECs 2010-R13 to -R20 and -R22
2010 Sept. 3: 19:05. It seems that the issuance of the MPECs
2010-R13 to -R20 and -R22 was affected by a disk error. The circulars
are being remailed. We apologise if you get two copies of these
circulars.
- Reuse of circular number "MPEC 2010-Q01"
2010 August 30: 09:30. The MPEC circular number 2010-Q01 was
inadvertently reused by the overnight DOU MPEC routine. It
should have been 2010-Q41. The problem was caused by a reorganization
of disk storage and an apparent failure to copy the latest version
of the MPEC index file. Last night's DOU MPEC has been
renamed 2010-Q41 in the on-line version.
- Network interruption August 12
2010 August 4: 12:23. We have received notice that Harvard will be
performing maintenance on a core router on Thursday, August 12,
05:00-07:00 EDT. There may be intermittent outages of connectivity
in this period.
- HTML-4.01 compliance
2010 July 11: 20:00. We have received a query about our use of the
HTML-4.01 compliance icons on most of our pages. If you test the
compliance you will see five errors reported. These errors are not
due to anything on our pages but are the result of a single line
of usage tracking code that is automatically appended to each
file returned by the server. The invalid nature of the line has
been reported to the hosting company, but they appear to be unwilling
to fix it. In the near future, the main MPC site will move back to
computers hosted at CfA. Some time thereafter, the mirror site
will be moved to a new, permanent location.
- Preprepared Ephemerides
2010 July 4: 21:45. The bulk of the preprepared ephemerides that used to
be available at /iau/Ephemerides have been restored. A new feature is
the ability to download KML files of the paths of the comets for display
in Google Earth.
- Last night's DOU MPEC
2010 May 31: 12:24. Last night's DOU MPEC has been abandoned as a
result of a system failure overnight.
- Web Server Unavailability
2010 April 12: 14:40. We are disabling the MPC webserver for an hour or
so to help us track down a network issue.
- 15:40. The MPC webserver has been restarted.
- Possible Network Disruption (March 10)
2010 March 9: 15:45. On Wednesday, March 10, starting at 09:00, we will be
reconfiguring our VMS cluster. For outside users the most visible sign
of this reconfiguration will be (hopefully) short outage of the MPC
webserver. Since the reconfiguration involves changes in our DNS
registration, it may take time for the changes to propogate outside
CfA.
- March 10: 14:00. The network reconfigurations have been completed.
The changes are visible internally.
- 17:10. As of now, the DNS changes are not visible from my home
system.
- 22:40. It is probable that the inability of outside use to access
cgi services of the MPC webserver is due to blocking of http traffic at
the sub-net level. This suspicion is apparently confirmed by the fact
that my home systems see the new IP address for the MPC webserver
This problem has been reported to the Computation Facility.
- March 11: 10:10. The cgi scripts are again visible externally.
- Official MPC Mirror Site II and Mid-Month MPS
2010 Feb. 18: 12:20. Efforts to get the MPC mirror site fully up and
functional are proceeding slowly. Progress will remain slow until such
time as access is restored to the disabled user accounts in the CF.
It is worth pointing out that it seems unlikely the CF-hosted MPC
website will be updated in the future. The reasons for this are
not relevant for public discussion, but we will agree if this is
the decision of the CF. All future updates will therefore occur only
on the
www.minorplanetcenter.org site.
Simply replace "www.cfa.harvard.edu" or "cfa-www.harvard.edu" with
"www.minorplanetcenter.org" and you should find the desired resouce
on the new site (eventually!).
We can make no guarantees at this stage as to when the next mid-month
MPS batch will be released. We will be looking at the
preparation procedure in the next few days.
- Official MPC Mirror Site
2010 Feb. 15: As a result of the hacker intrusion mentioned in Feb. 12
note, we are bringing the
official MPC mirror site on-line immediately.
Please understand that this early release is solely to make certain
data available to certain professional collaborators. The completeness
of the mirror site is currently minimal and the current unavailability
of access to the main site means that it is not trivial to copy the
necessary data to make it complete. In addition, most of the routines
that copy data to the main website are not yet modified to copy data
to the mirror site.
- Successful hacker intrusion at CfA
2010 Feb. 12: 11:00. We have been informed by our Computation Facility
that the CF account of a MPC staff member has been hacked. Consequently,
that account has been disabled, as has the network access to the MPC's
Linux cluster. At this point, we have no evidence that the core VMS
cluster was compromised.
The upshot of this hack is that updating of the MPC website is not
possible and circulars cannot be mailed out via their normal route.
We have set up alternate routes for mailing circulars, but recipients
who check the From: line of messages before allowing delivery of the
message will probably not receive the circulars. There is nothing we
can do at this end about this problem. It is also unclear when we will
be able to update the webpages.
Static pages such as the MPES main page remain functional, but
periodically-updated pages such as the NEOCP do not. Observers are advised to
use the mirror
page links.
- SMTP E-mail Interruption (January 29)
2010 Jan. 26: 16:57. We have been informed that the CF's SMTP server
will be off-line for approximately 30 minutes beginning at 06:05 EST
on Friday, January 29, to allow OS patches to be installed. Attempts to
send e-mail to us during this period will fail, but the mail should
queue up on the sending computer for a subsequent resend attempt.
- Web Services
2010 Jan. 2: 18:30. The web services are functioning, but the
webserver is getting pounded by MPChecker request. We will
examine the logs and, if the requests are coming from the same
user, will block access from their IP address.
- Internal Network Problems II?
2010 Jan. 1: 14:00. There appears to be a problem with network
connectivity. It is unclear at this point whether there is
any relation to the Dec. 30 problems. The problem is causing
the web services to hang. We are investigating.
- Jan. 2: 11:15. Attempts to resolve the problem remotely were
unsuccessfully (although we believe we know where the problem
originated). We have just arrived in the office and will be attempting
manual fixes shortly.
- Jan. 2: 12:00. We believe we have fixed the problem and are checking
the cluster status and the state of the numerous automated processes that
handle the bulk of the MPCs work. Normalcy should be restored shortly.
- Internal Network Problems?
2009 Dec. 30: 21:55. There was a hang of (at least part of) the CfA's
internal network from about 18:44:15 to 21:29:30. All high-level
network protocol traffic between the MPCs computers froze. Low-level
protocols continued to function allowing the MPCs cluster to remain
intact. It is very probable (though we didn't actually check) that
some or all of the web services were off-line during this period. We
have queried this event with the Computation Facility but
have not yet received a response.
- Problem with MPCOBS
2009 Dec. 19: 18:50. Two independent reports have been received of
errors (specifically, stack dumps) being returned by the MPCOBS
service. We are investigating.
- 19:15. We've located the problem and fixed it. We've
also made the error message more "friendly" in case a similar
problem arises in the future.
- Maximum version limit reached in MPES
2009 Oct. 21: 10:00. Some debugging output that was added in order to fix
the Oct. 1 problem was left in. This caused one debugging output file
to reach the maximum allowable version number. This code has now
been deactivated.
- Missing uncertainty information in MPES pages
2009 Oct. 11: 01:40. A problem that caused the uncertainty information to
be omitted from MPES pages returned by the MPEph.COM script has been
fixed. This problem first appeared when the Oct. 1 problem was fixed.
- Strange behavior of MPC webserver
2009 Oct. 1: 20:30. We have noticed strange behavior of the MPC webserver
today. On five occasions, cgi jobs started executing but then simply
hung. Once the maximum number of allowable processes are in the hung
state the webserver stops responding to requests. We are investigating.
- Oct. 3: Debugging/logging output has been enabled in the command file
that calls the executable, the C front-end of the executable and the
Fortran code. The jobs that are stalling create log files from the
command file and the C front-end, but the Fortran log files, although
created, are empty. The reasons for this are still baffling.
- Oct. 3: 23:45. We've found the reason the script is hanging. An
array used by a library routine is being overwritten. The reason why
this overwriting is occurring is not yet clear. Particularly why it
has only surfaced in the past few days.
- Oct. 4: 00:30. We believe that the hanging script problem is now
fixed and we understand why the problem appeared suddenly a few
days ago.
- Free access to previously-restricted MPC data
2009 Oct. 1: 00:20. The steps necessary to remove the subscription blocks
to previously-restricted MPC data have been started. An issue has arisen
that requires a fresh mind. The blocks will be removed in the morning.
Please note that access blocks will be put in place when we are updating
the files or when we need to do other work on the pages.
- MPEC e-mail problems
2009 Sept. 30: 09:00. A problem with a number of recent MPEC mailings
has been identified that caused many (all?) outside subscribers to not
receive the circular. The underlying immediate problem has been fixed
and a procedure has been modified to try and prevent this happening again.
The problem affected only the three most recent circulars.
- Incomplete files on the MPCORB anon-ftp site
2009 Sept. 29: 20:30. A user reported that a specific file on the
MPCORB anon-ftp site was incomplete. A check showed a large number
of blank files. We are checking the logs.
- 20:40. A look at the OpenVMS-side logs shows that part way
through the copying of the files the error message "write failed"
started appearing. No further information was returned by the sftp
system, so it is unclear where the problem lies. It is unlikely that
the available disk space dropped to zero (the ftp directory sits on
a 3 TB device and there are currently 585 GB free). We are running
the copy routine manually. We will check the situation after the next
DOU MPEC.
- 21:05. Some of the zero-length files have been fixed. The remaining
problem cases will have to await the issuance of the next DOU MPEC.
- Problem with NEOCP comments
2009 Sept. 28: 11:45. A number of users reported problems with the latest
version of the NEOCP comments cgi script rejecting all comments. An
updated executable has been put in place.
- CfA anon-ftp disk full
2009 Sept. 19: 14:15. The disk used for storage of the files on the the
anon-ftp server (e.g, the MPCORB files) is full. Until space is cleared
updates of MPCORB will be incomplete.
- Accessing mpn.arc via browsers
2009 Sept. 8: 10:40. Since the mpn.arc file available for download in the
ECS is now over 4 GB in size, a number of browers fail to download the
entire file. That these browsers are limited to 32-bit filesizes is
clear from the fact that the partial downloads consist of filesize-4GB
bytes. Until the affected browsers can cope with >32-bit filesizes,
this file will have to be downloaded via ftp.
- Inability to access cfa-ftp.harvard.edu
2009 Sept. 6: 17:30. An outside user reported his inability to access the
MPCORB files on cfa-ftp.harvard.edu. We can confirm that there seems
to be a problem with ftp access as we see the same problem from our
home machines. We have queried this issue with the Computation Facility
(of course, it's a long holiday weekend...).
- Sept. 7: 14:00. It seems that the outage was intermittent. Access
to the MPCORB files on cfa-ftp is again possible.
- Network Interruptions (September 4)
2009 Aug. 26: 17:00. We have been informed that there will be a network
outage from 17:15-21:00 EDT on Friday, September 4. The CF will be
upgrading core networking equipment. The whole complex will be
off-line during the upgrade.
- SMTP E-mail Interruption (August 7)
2009 Aug. 5: 14:00. We have been informed that the CF's SMTP server
will be off-line for approximately 30 minutes beginning at 06:05 EST
on Friday, August 7, to allow OS patches to be installed. Attempts to
send e-mail to us during this period will fail, but the mail should
queue up on the sending computer for a subsequent resend attempt.
- MPN.ZIP File In MPCAT-OBS
2009 July 9: 17:50. The procedure that builds the zip'ed version of the
MPN.ARC file in the ECS service MPCAT-OBS has failed as MPN.ARC is now
over 4 GB in size. We had a similar problem ~ 2.5 years when the
filesize grew over 2 GB. We are attempting to use gzip to compress the
file. The file extension will remain .zip for the forseeable future.
- Possible Disruptions (July 11)
2009 July 1: 11:26. We have been informed that there will be a scheduled power
shutdown at a remote CfA site on Saturday, July 11, from 07:00 to
13:00. A number of CF machines are located at this site and they
will begin to be powered down at 16:00 on July 10. It is unknown at
this point if this outage will affect internal MPC operations or
external services.
- Network Interruptions (July 9)
2009 June 30: 16:12. We have been informed that Harvard will be doing work
on a core router from 05:00 to 07:00 on Thursday, July 9. This will
cause the CfA to be off-line for periods during the work window.
- Network Outage
2009 June 28: 22:00. All connections to/from the CfA network failed around
15:35 when a router at Harvard failed. Connectivity was restored
several hours later.
- June 26 DOU MPEC
2009 June 26: 11:15. It seems that the fall-out from the June 24 problem
continues. Two versions of last night's DOU MPEC were partially prepared.
The correct circular number should be 2009-M38, but the version numbered
2009-M39 was mailed out. Neither process completed. We are fixing the
bits that failed.
- 12:15. It has been decided to redo the preparation of last night's
DOU MPEC. It was simply too fiddly to fix just those bits that failed.
The DOU MPEC will be MPEC 2009-M38.
- Mis-mailing of old DOU MPEC
2009 June 25: 11:05. The problems overnight with access to the CF cluster
caused a script running there to become "confused". It proceeded to
send out MPEC 2008-F58, rather than MPEC 2009-M35. We (think we) have
killed the processes that are mailing out the old circular, but a large
number of subscribers may receive MPEC 2008-F58.
- MPEC Issuance and Website Updating
2009 June 24: 22:17. We are currently unable to issue MPECs,
apparently a result of the OS upgrades mentioned in the item below.
We have no automated transfer of files from our cluster to the CF
webserver or the CF cluster. The problem also affects the automated
updating of all our pages that are hosted on the CF webserver.
- June 25: 09:30. The problem was apparently cleared some time
this morning. We were not notified that it was fixed.
- CF Webserver Outage on June 24
2009 June 24: 12:17. We have been informed that the CF will be doing
OS upgrades on the NetApp filers which, amongst other things, serve
up the CF webpages. The CF webserver will be
off-line during this period. The upgrades will start between 17:15 and
17:30 and are expected to take 45 minutes. Access to MPC cgi
scripts will continue to be possible during this outage via
our mirror
pages.
- URLs in the "New Look"
2009 June 18: 12:45. It is worth remarking that, except as noted below,
all URLs in the old look MPC website continue to work unchanged in the
new look. The exceptions are:
- The MPC Status reports covering the period 2003-2004, which were
previously stored in monthly files, have been amalgamated into six-monthly
files. This cuts down on the number of files.
- Checker Services
2009 May 15: 21:20. A user reported that MPChecker failed to locate a
recently-designated NEO. The object was missing from the datafile
used for the NEO/NEOCMT/MPChecker services. The problem has been
traced to a disk that is used by the procedure that generates the
special-epoch elements needed for these services getting filled up.
The problem has apparently existed for the past three days. A slight
addition to the procedure has been made to try and prevent this happening
in the future. The affected files are being regenerated.
- 21:38. The "problem" object mentioned by the user is now found
by NEOChecker.
- Webserver
2009 May 6: 10:00. We have received a number of reports about perceived
problems with the MPC webserver overnight. The webserver has been
functioning normally, it was simply being swamped by a very large number
of MPChecker requests from a machine at ESO.
- Non-emailed MPECs
2009 May 2: 11:00. Two subscribers have reported non-delivery of MPECs
2009-J04 through -J07. Later MPECs were delivered without problem.
It seems that the machine that mails out the circulars was rebooted
(probably to do patch installation) around the time that the
non-delivered circulars were issued. The mailing of these circulars
is being redone. Some subscribers who are at the start of the mailing
lists may thus receive a second copy. We can't easily avoid this
problem.
- AUTOACK Problem Overnight
2009 April 22: 10:40. A problem with an observer submitting some
previously-reported observations caused AUTOACK to stall around 02:43
this morning. AUTOACK kept claiming that the stalled batch had been
removed, but it never got cleared. The procedure has been modified to
prevent this happening again.
- Lists of NEOs and TNOs
2009 March 18: 22:00. We are aware that there are some problems with the
program that generates the lists of NEOs and TNOs on our site. A new
version of the program is in preparation and we expect it to be on-line
by early next week.
- SMTP E-mail Interruption (March 12)
2009 March 10: 11:20. We have been informed that the CF's SMTP server
will be off-line for approximately 30 minutes beginning at 06:05 EST
on Thursday, March 12, to allow OS patches to be installed. Attempts to
send e-mail to us during this period will fail, but the mail should
queue up on the sending computer for a subsequent resend attempt.
- MPC Webserver Problem
2009 Mar. 9: 16:00. The MPC webserver is experiencing hangs for an unknown
reason. Stopping and restarting the webserver clears the problem but
within minutes the webserver stops serving pages. We are investigating.
- Mar. 10: 12:10. It appears that the hangs are being caused by
"resource wait for system communication services" (RWSCS). An obvious
cause of this is errors in cluster communications caused by failing
intranet components. Investigations are continuing.
- 14:30. The problem is proving hard to track down. It is possible
that it related to one of the cgi scripts and the fact that the Minor
Planet Centre is in "preparation" mode. We should go back to "processing"
mode late tonight. If the problem disappears at that time, it will
give us a bit hint that the afore-mentioned cgi script is the culprit.
- 10:30. The MPC went back to "processing" mode in the early morning.
The MPC webserver has been stable since late last night.
- CF Webserver Outage on Feb. 26
2009 Feb. 19: 12:17. We have been informed that the CF will be doing
OS upgrades on the NetApp filers which, amongst other things, serve
up the CF webpages. The upgrades will start between 17:15 and
17:30 and are expected to take 30-40 minutes. Access to MPC cgi
scripts will continue to be possible during this outage via
our mirror
pages.
- Issue with Missing Orbits During MPC Preparation
2009 Feb. 8: 11:45. An alternate fix for the long-standing problem with
elements for object that are being numbered being unavailable in the
MPES for a period during MPC preparation has been found and implemented.
- Funding for MPC restored
2009 Feb. 3: 13:00. Funding for the MPC has been restored. The previously-posted
statement on this topic (dated Jan. 15) has been removed. The files associated with the January
MPCs are being posted right now.
- Version Number Overflow
2009 Jan. 7: 11:00. A number of cgi scripts started failing overnight when
they could no longer produce temporary output files due to those files
reaching the maximum version number. These temporary output files
were used for debugging and are no longer needed. They are being
deleted.
- Recent MPECs
2009 Jan. 2: 13:23. Recent MPECs (i.e, those issued after
MPEC 2009-A05) are not accessible due to a protection problem
on the directory containing the 2009 circulars. We are attempting
to fix the problem.
- 13:44. We believe the protection problem is fixed and the missing
circulars have been replaced.
- Missing Objects in MPChecker
2008 Dec. 30: 17:50. A number of users have reported inconsistent output
from MPChecker when checking more than one object. It seems that some
code, recently added to remove duplicate objects from the output list,
is removing objects duplicated in different fields. A fix is being
worked on.
- 19:00. A fix is now in place.
- No NEO Observations on DOU MPECs
2008 Dec. 23: 11:30. Recent DOU MPECs have not contained any new
new NEO observations due to a problem with the routine that extracts
the observations. A fix is in place. The missing observations should
appear on the next DOU MPEC.
- Erroneous Designations in MPChecker
2008 Dec. 21: 10:00. A user reported that certain objects in MPChecker
are not correctly labeled, specifically that unnumbered objects are
labeled erroneously with numbered designations. The positions and
motions are correct, but the designation is wrong. The problem seems
to be restricted to numbered objects above (199999). We are investigating
the problem.
- 12:00. As suspected, the problem lay with the labeling of orbits
for numbered objects above (199999). The problem is now fixed.
- Issue with Missing Orbits During MPC Preparation
2008 December 11: 11:45. There has been a long-standing problem with
elements for newly-numbered objects being unavailable in the MPES for a
period
during MPC preparation. This has been a troublesome problem
to investigate as the problem only occurs while we are busy with the
filing of the new data, meaning we do not have time to examine the
issue. We have now found a few free moments, have identified a possible
source for the problem and put a fix in place. We will see if the
problem disappears during the next MPC preparation period.
- 15:15. Unfortunately, it appears that the above fix is causing
severe performance problems for the MPES. An alternate fix will have
to be found. The MPES executable has been restored to its previous
version.
- E-mailing of circulars
2008 December 3: 10:25. The e-mailing of circulars issued since noon
yesterday has been interrupted by problems caused by transfer of the
mailing scripts from a Solaris machine that has been retired to a Linux
box.
The problems are being examined and a fix should be forthcoming shortly.
The backed-up circulars will then start being delivered automatically.
- 10:53. We believe that the problem (related to interpretation of
relative pathnames between Solaris and Linux) has been fixed. The
backed-up circulars are being delivered.
- Anon-ftp disk full
2008 November 27: 11:18. The disk on which the anon ftp files (such as the
MPCORB datafiles) reside filled up overnight.
- November 28: 09:15. The anon-ftp disk again filled up overnight.
- December 2: 09:15. Someone has cleared space on the anon-ftp disk
as it is currently 70 percent full.
- Mid-month observation files in ECS
2008 November 24: 11:00. A number of subscribers have reported that they
cannot access the files in /ECS/MPCAT-OBS/midmonth/. It seems that
this is related to the CF machine changeover mentioned below. We
are investigating, but we have no information of what actually got
executed on the Linux box.
- 12:15. Some manual digging shows that a cron job script (to build
the zipped versions of files in the /midmonth directory) did not
run. This script also clears the lock that prevents users accessing
the directory while the zipping is being performed. The script is
being run manually. The cron job problem remains under investigation.
- 13:05. We believe the problem is fixed.
- SMTP E-mail Interruption (November 24)
2008 November 19: 17:02. We have been informed that the CF's SMTP server
will be off-line for approximately 30 minutes beginning at 06:05 EST
on Monday, November 24, to allow OS patches to be installed. Attempts to
send e-mail to us during this period will fail, but the mail should
queue up on the sending computer for a subsequent resend attempt.
- ECS files for November MPC
2008 November 17: 12:50. The ECS files for the November MPCs are
currently being posted. A large number of problems related to the
Solaris-Linux transition of the CF machine (see earlier notes) have
arisen. Users are warned to expect problems...
- DISCSTATUS reports
2008 November 15: 20:57. The naming status of numbered minor planets
with numbers above 200000 is incorrect in the just-mailed DISCSTATUS
reports. The count of oppositions may also be incorrect on some
objects. The problem has been fixed and the next run will correctly
flag these objects.
- ECS files for November MPC
2008 November 15: 10:10. Due to a rather knotty problem connected to the
preparation of the PDFs of the three sets of circulars, the ECS files
for the November MPCs will not be posted until late on Monday,
November 17.
- Anon-ftp disk full
2008 November 15: 00:40. The disk on which the anon ftp files reside has
again filled up. We have reported the problem to the CF.
- Unavailability of CF Webserver
2008 November 7: 20:00. The CF webserver is not serving up pages. The
problem has been reported to the CF. Our cgi scripts continue to
be accessible through the
mirror page.
- November 9: 00:39. We've just noticed that the CF webserver appears
to be operational again. There was no notification from the CF.
- Stale Information in MPES
2008 November 6: 00:45. We have noticed that some information in the MPES
is stale. The affected information is that derived from external sites
and which is supposed to be copied from those sites on a regular basis
via a number of Python scripts. Examples of such data files include lists
of Arecibo and Goldstone radar targets. In some
cases, the MPES is working from data files derived from files downloaded
in September. This problem predates the Oct. 16 problem with a CF
server, but the problem is OS related: the Python exectuable was moved
from /opt/bin to /usr/bin. We believe all affected scripts have been
modified and data files should become current after the next run of the
relevant script.
- E-mail Delivery of MPECs
2008 October 30: 11:30. We have received a number of reports from subscribers
who have not received recent MPECs. We have received our copies
of recent circulars, so the problem would seem to be restricted to
some/all external addresses. The machine that sends out the circulars
is a CF machine. We are investigating...
- 16:40. We are resending MPECs U92-U100.
- 16:55. The first attempt used malformed scripts. We are trying
again... Note that you will receive the resent DOU MPECs
even if you've requested non-delivery of these circulars. We apologize
for this minor inconvenience.
- Network Interruptions (November 4)
2008 October 29: 12:40. We have been informed that the Computation Facility
will be performing software upgrades on core network equipment on
Tuesday, November 4, from 06:00 to 07:30 EDT. We expect MPC (and other
CfA) webpages to be unavailable during this period. Access via the
mirror
pages will also be affected.
- MPCORB files and mirrors
2008 October 20: 12.56. The updating of the MPCORB datafiles on our
site and the mirrors has been affected by two related problems.
The first is the previously-mentioned OS change on a CF machine.
The second is the failure to copy files to the mirror due to
a change in the host key. We are working on transferring new
host keys to the remote site.
- Latest mid-month MPS batch
2008 October 19: 11.15. Last night's mid-month MPS batch failed to
update correctly due to problems with the new CF machine mentioned
previously. We will have to manually reconstruct the bits that failed,
while modifying the scripts so that they work next time.
- 18:15. We believe all the failed bits have been fixed.
- October 20: 11.41 A user reported that the midmonth/mpu.* files
are missing. That bit of the update procedure is being rerun.
- 12:15. The midmonth/mpu.* files have been restored.
- Network Interruptions (October 21)
2008 October 16: 18:30. We have been informed that Harvard
will be upgrading core network equipment on Tuesday, October 21, from
06:30 to 07:30 EDT. We expect MPC (and other CfA) webpages to be
unavailable during this period. Access via the mirror pages will also
be affected.
- Power outage at Observatory II
2008 October 16: 12:00. Yesterday's power outage fried one of the CF
servers. Unfortunately, it was the machine on which many of the
cron jobs run that handle data transfer between the MPC and the CF
webserver. A new machine is in place, but the cron jobs will not
be restored until early next week. We will try and replace as much
as we can, but some web pages and data files may not get updated
as expected.
- October 18: 12:00. Some of the cron jobs are working again, but
others continue to cause problems. This is a result of the old machine
being a Solaris box while the new machine is a Linux box. Specifically,
the preparation of
the CR-terminated versions of various MPCORB datafiles is failing due
a segmentation fault. Also, the copying of files to the official
mirror site is failing due an incompatability in specifying batch
mode in the command-line flags: the original version of sftp used
-b to specify batch mode; an "upgrade" to sftp caused batch mode to
be specified with -B; the version on the new Linux machine again
uses -b! We believe that all cron jobs that use sftp have been modified
to reflect the flag usage de jour.
- 18:00. The October Minor Planet Circulars will not be
published, partly due to the power outage and partly due to staff
absences.
- Power outage at Observatory
2008 October 15: 12:51. An unexpected power outage affected the Observatory
(and, apparently, parts of Cambridge) around 10:50 today. The MPC
machines began shutting off 15 to 20 minutes later when they exceeded
the capacity of the UPSes. The power has just been restored, the
cluster has reformed and processing routines are being restarted.
Normalcy should be resumed shortly.
- 13:15. While the MPC's systems are fully functional at this point,
it seems that (at least some parts of) the Computation Facilities are not.
We are not receiving e-mail at this point and we cannot access CF-hosted
machines. We expect these problems to disappear shortly.
- 14:20. E-mail from outside is beginning to arrive.
- 23:55. There were problems with the MPC webserver. The webserver
was restarted.
- MPEC 2008-T101
2008 October 8: 17:15. Due to a production error (two people working on
announcing a new object at the same time), the object 2008 TK9, which
appeared on MPEC 2008-T100, was also announced as 2008 TL9 on MPEC
2008-T101. The latter designation and MPEC have been retracted.
The next MPEC to be issued will be 2008-T101.
- Linking of designations in MPES
2008 October 8: 01:25. A user reported that entering the non-principal
designation 2008 SC7 into the MPES did not return the elements or
ephemeris
for the principal designation 2004 XX145. The problem, which presumably
affected other objects as well, was traced to a
malformed entry in the new double designation file and fixed.
- No space left on on ftp server
2008 September 28: 15:00. The anon ftp server disk is again full (currently
only zero bytes are free). Updating of various files stored there may
fail until such time as space is available.
- No space left on on ftp server
2008 September 19: 14:27. The anon ftp server disk is again full (currently
only 4 KB are free). Updating of various files stored there may
fail until such time as space is available.
- Corruption of files on ftp server
2008 August 28: 11:13. It has been reported that the MPCORBcr.DAT file
is corrupted. Investigation shows that a number of files in the
pub/MPCORB directory are blank or incomplete. There is an error
message in the logfile of last night's DOU MPEC preparation,
but it isn't very helpful.
- August 29: 11:08. The problem is still present after last night's
DOU MPEC. The same cryptic, unhelpful error message is present
in the DOU preparation logfile.
- 11:17. It seems that the disk on which the anon FTP data sits is
full.
- 14:00. We have cleared over 1 GB of no-longer-needed material from
the anon-ftp disk.
- 21:30. As of now, there are only 8 KB free on the anon FTP disk.
Tonight's update is therefore likely to fail and since it's a long
holiday weekend in the U.S. space may not get freed up until the middle
of next week. Unfortunately, this is a disk that we do not control.
- 22:10. Some other user has apparently cleared up old files, as
there is now 1.1 GB free. Of course, this free space may disappear before
the DOU MPEC procedure runs...
- Designation Message Problem
2008 July 28: 12:50. It has been noted that some observers are receiving some
incorrect correspondences between their temporary designations and
MPC designations. These designations are coming from a reprocessing of
the old unidentified material. The problem occurred because the
reprocessing is being handled in a different mode than the normal
processing. Further, this problem affects only the messages
sent to the observers. A fix to avoid this confusing problem has been
implemented.
- SMTP E-mail Interruption (August 4)
2008 July 22: 16:45. We have been informed that the CF's SMTP server
will be off-line for approximately 60 minutes beginning at 06:00 EDT
on Monday, August 4, to allow OS patches to be installed. Attempts to
send e-mail to us during this period will fail, but the mail should
queue up on the sending computer for a subsequent resend attempt.
- CF Webserver Outage
2008 July 8: 23:33. The Computation Facility's webserver has apparently
died as attempts to access any of our pages (or the SAO main page)
all time out. The problem was reported to the CF at 23:32:12 EDT.
Note that access to various observer services via the mirror
pages was unaffected by this problem.
- July 9: 09:00. The CF reports that the webserver has been restarted.
- SMTP E-mail Interruption (July 7)
2008 July 3: 13:13. We have been informed that the CF's SMTP server
will be off-line for approximately 30 minutes beginning at 06:00 EDT
on Monday, July 7, to allow OS patches to be installed. Attempts to
send e-mail to us during this period will fail, but the mail should
queue up on the sending computer for a subsequent resend attempt.
- Archive of older enhancements and problems/resolutions