TSD Operational Log - Page 4
Starting 2022-09-05 15:00, we'll migrate approximately 1000 projects from the old to the new storage solution. Affected projects will have a downtime from 15:00 until 15:00 the next day, unless specified otherwise in the notification email. For more information about the migration see our FAQ.
Update: Not all projects were migrated by 15:00. The downtime for these projects has been extended.
The Software Center on all Windows VMs will be unavailable between 09:00 and 16:00.
As of 2022-08-15 10:30, TSD Self Service is experiencing technical difficulties. This issue affects various operations pertaining to performing updates.
We are working on a solution.
Project Database machine will receive a minor DB security upgrade.
Database Upgrade will disrupt login from 9AM till 10 AM Friday 15.07.
Affected services: Selfeservice, File Transfer and Logging. We apologize for the inconvenience. We will work to make the process as smooth as possible.
We are doing network maintenance in TSD Monday-Tuesday 23/05/2022-24/05/2022.
No downtime is expected but interruptions may occur.
The matlab license on Linux hosts expired. We are working to solve the problem as quickly as possible.
Colossus will have downtime Wednesday 2022-03-16 from 08:00-16:00. We have set a maintenance reservation on the Slurm controller for this period.
Any jobs that cannot be scheduled and completed before the downtime will remain pending and report "ReqNodesNotAvail, reserved for maintenance", until after the downtime, when they will be rescheduled and resumed automatically.
During the downtime we advice you to keep an eye on this operational log for any updates.
Login to TSD is currently unavailable.
We are working to solve the problem as quickly as possible.
Our apologies for the inconvenience.
--
The TSD Team
Consent and Publication internal portals are not available. Investigation is in progress …
The Slurm controller on Colossus was unresponsive between approximately 01:00 and 09:15 on February 17th.
The issue has been resolved, but we advice users that had running jobs during this time frame to check their job logs for issues.
We will be performing an upgrade of the Horizon view infrastructure on Thursday February 3rd at 12:00. We expect this to take no more than 2 hours. We will update this message and send out an email once the upgrade is complete.
Login through view.tsd.usit.no will be down during the downtime. The project machines will not be affected, so any existing sessions you have will still be there when you log back on after the downtime.
Due to a software error, the Windows access of some users was extended past the period to which the project Administrator has approved. Now this is corrected. If you have lost your Windows machine access, please contact your project administrator to add you to the appropriate group.
f4transkript is currently not available in TSD-projects.
We are working to solve the problem as quickly as possible.
Our apologies for the inconvenience.
We will be performing network maintenance on Monday 13/12/21 at 12:00, the downtime will last 10-15mins.
We do not expect any downtime, but there might be some interruptions.
Between approximately 10:00 and 10:30, the module load command did not work on the submit hosts and Colossus nodes. Please resubmit any jobs that may have failed. This has been resolved.
We will be performing maintenance on view-ous.tsd.usit.no on Tue 16/11 12:00.
The downtime will last about an hour.
BankID is having planned maintenance downtime on Thursday 2021-10-21 00:01–08:01, and will not be usable for authentication while this work is ongoing.
For TSD, this means that login to TSD Selfservice via BankID and authentication to Nettskjema forms using BankID will be unavailable in the above specified timeframe.
For further information please see Digdir's downtime notice:
https://status.digdir.no/incidents/pnlr6gn69dy4
-----
Update: BankID for mobile is working again. BankID with “kodebrikke” is still unavailable. This will also be unavailable on Oct 23
Login to TSD is currently unavailable.
We are working to solve the problem as quickly as possible.
Our apologies for the inconvenience.
--
The TSD Team
Update 20:30: Maintenance on the ESS storage system has been completed and the submit hosts are available again. Some submit hosts have been rebooted.
Update 18:58: Computation on Colossus has been resumed.
Update 18:48: The downtime has been extended, since the final update will result in additional NFS hangs.
Update 16:20: The downtime will be extended by approximately 2 hours. There may be additional NFS hangs on hosts mounting /cluster during this period.
Update 16:00: The last few Colossus nodes are being upgraded. Colossus will resume after that.
Colossus will have downtime on Wednesday 13/10 due to third-party maintenance on the /cluster file system. The affects /cluster on Colossus as well as NFS mounts of /cluster on the submit hosts.
Submit hosts will experience intermittent NFS hangs across the day.
Any jobs submitted to Colossus that cannot completed before the scheduled downtime w...
Hosts mounting /cluster/projects/pXX and /cluster/software experienced several NFS hangs on Saturday 09/10 and Sunday 10/10. User processes accessing these shares were killed upon remounting the NFS mounts.
The issue has been resolved.
We will be performing maintennance on the horizon view login servers, the downtime is estimated to last 1 hour.
https://view.tsd.usit.no and https://view-ous.tsd.usit.no will be unavailable during the downtime.
--
The maintennance is now complete.
Nettskjema will no longer automatically deliver encrypted JSON-files to all TSD-projects.
This will not affect the use of Dataloader.
If you still want to recieve encrypted JSON files for each submission, please notify tsd-drift@usit.uio.no
Colossus will have downtime on Wednesday 13/10 due to third-party maintenance on the /cluster file system. The affects /cluster on Colossus as well as NFS mounts of /cluster on the submit hosts.
Any jobs submitted to Colossus that cannot completed before the scheduled downtime will remain pending until after the downtime. They will be rescheduled automatically once the downtime ends.
During the downtime we advice you to keep an eye on our operational log for any updates.