Georgia Tech IT
Active Incident

Updated a few seconds ago

PACE - Phoenix Scheduler OutagePartial Service Disruption

Incident Status

Partial Service Disruption

Components

Academic Services

Services

PACE


May 13, 2024 2:54PM EDT
[Identified] Intermittent failures of the Phoenix scheduler.

May 13, 2024 2:33PM EDT
[Identified] We are experiencing intermittent failures of the Slurm scheduler on Phoenix. Jobs may start but fail to actually run, experience longer-than-normal completion times, encounter other failure modes. We are investigating the issue and will update as we learn more.
Barracuda WAF issue affecting Plesk HostingDegraded Performance

Incident Status

Degraded Performance

Components

Web Hosting

Services

Plesk, WAF - Web Application Firewall


April 24, 2024 8:47AM EDT
[Monitoring] OIT is continuing to monitor the situation, and there have been no significant updates yet. Support teams are still working with the vendor to get a more permanent fix to the issue.

April 16, 2024 3:35PM EDT
[Monitoring] While the workaround to address the WAF issue on the the Plesk Webhosting complex on 4/5 did restore service, it unfortunately also is causing an IP forwarding issue which is affecting the functionality of some of the sites. Support is continuing to work with the vendor to restore the WAF to service so that the webhosting complex can be brought back to its original state.

April 12, 2024 2:51PM EDT
[Monitoring] The workaround that was put in to restore connectivity to plesk hosted websites continues to be working. However, a permanent solution to the underlying causing is still being investigated and worked on with the vendor. Once a working solution is found, tested, and implemented, this issue will be cleared . Until then, we will continue to run off the workaround, and support is continuously monitoring the situation for any new issues that could spring up from the workaround.

April 6, 2024 12:34AM EDT
[Monitoring] The workaround was successful, and sites should be accessible once again. Support is continuing to work with the vendor to address the underlying issue for a more permanent fix.

April 5, 2024 11:33PM EDT
[Identified] Support is continuing to work with the vendor to try and fix the issue with the WAF. However, network team is currently getting ready to implement a workaround to restore connectivity to the affected sites as a temporary measure until a fix for the WAF issue can be found. Once the workaround is implemented, approximately 30 minutes of down time on the hosted sites is expected.

April 5, 2024 1:51PM EDT
[Identified] Staff are still working with the vendor to resolve the WAF issue. If students need to access Canvas, they can reach it by going to https://gatech.instructure.com/

April 5, 2024 11:06AM EDT
[Identified] The vendor's patch was unsuccessful in resolving the issue and will rollback their patch. Efforts are being made to prevent any downtime, but it is possible that there may be a brief outage of up to 10 minutes. Support is continuing to work with the vendor to troubleshoot the problem.

April 5, 2024 9:23AM EDT
[Identified] The vendor is applying a patch to address the communications issues with the Web Application Firewall. Efforts are being made to prevent any downtime, but it is possible that there may be a brief outage of up to 10 minutes.

April 5, 2024 8:33AM EDT
[Identified] Staff have discovered a communications issue between the Plesk hosts and the WAF (web application firewall) that is causing sites hosted on Plesk to be slow or unreachable. Staff are continuing to work with the vendor to resolve this issue. Sites affected include but are not limited to: buzzport.gatech.edu canvas.gatech.edu crc.gatech.edu ctl.gatech.edu dining.gatech.edu diversity.gatech.edu finance.gatech.edu honors.gatech.edu hosting.gatech.edu housing.gatech.edu ipat.gatech.edu

April 5, 2024 8:00AM EDT
[Investigating] Several sites hosted on the webplesk are unreachable due to issues with the WAF (web application firewall) including but not limited to : buzzport.gatech.edu canvas.gatech.edu crc.gatech.edu ctl.gatech.edu dining.gatech.edu diversity,gatech.edu housing.gatech.edu ipat.gatech.edu Staff are continuing to investigate with the vendor to resolve this issue.

April 4, 2024 4:58PM EDT
[Investigating] Several sites hosted on the webplesk are unreachable due to issues with the WAF (web application firewall) including but limited to : canvas.gatech.edu crc.gatech.edu ctl.gatech.edu dining.gatech.edu diversity,gatech.edu housing.gatech.edu ipat.gatech.edu

April 4, 2024 2:29PM EDT
[Monitoring] Service is currently restored and support is monitoring the situation. If you are continuing to experience issues, or if you have questions or concerns, please file a ticket via services.gatech.edu or contact the Administrative Services Center at 404/385-1111.

April 4, 2024 11:50AM EDT
[Identified] Related to the previous issue, web-plesk53 is now serving incorrect SSL certificates to sites hosted on the server. Support is working to resolve the issue.

April 4, 2024 9:49AM EDT
[Monitoring] Several sites hosted off of web-plesk53 were unreachable due to issues with the WAF (web application firewall) including: -Housing.gatech.edu, textbooks.math.gatech.edu, -undergradresearch.gatech.edu, -faset.gatech.edu, -transitionprograms.gatech.edu, -policylibrary.gatech.edu, -policies.gatech.edu, -alan.ece.gatech.edu -gtcad.gatech.edu -hasler.ece.gatech.edu Service is currently restored and support is monitoring the situation.

Academic Services

Partial Service Disruption

Campus Services

Operational

Campus Services - ITG

Operational

Campus Audio Visual Services

Operational

Email and Calendaring

Operational

Identity

Operational

Network

Operational

Web Hosting

Operational

Enterprise Data Services

Operational

IT Service Management

Operational

VideoConferencing

Operational

GTPE

Operational

Cloud Storage

Operational

External Services

AWS CloudFront

AWS EC2

Azure

Box

DocuSign

Dropbox

Duo

Honorlock

Instructure (Canvas)

Kaltura

LastPass

Let's Encrypt

MS Office365

MuleSoft

Salesforce

Sectigo

Zoom

Scheduled Maintenance

Email service maintenance - SPF update to Fail for OIT email domainsPlanned Maintenance

Schedule

May 27, 2024 7:00AM - 9:00AM EDT

Components

Academic Services, Campus Services, Campus Services - ITG, Campus Audio Visual Services, Email and Calendaring, Identity, Network, Web Hosting, Enterprise Data Services, IT Service Management, VideoConferencing, GTPE, Cloud Storage

Services

Active Directory, Top Level Web - AWS, Azure Active Directory, Banner and Oscar, BigIP and GTM, Box, BuzzCard - 3rd Party Retail Merchants, BuzzCard - Card Production, BuzzCard - Copier Readers, BuzzCard - Laundry Readers, BuzzCard - Online Deposits, BuzzCard - Pay for Print, Buzzport, Cable TV (GTCN), Campus Network, Canvas, Cloudbolt, Contract Information System, Centralized Endpoint Backups, ctr.gatech.edu, DegreeWorks, Deltek Costpoint, DHCP, Digital Room Indicator, DNS, Door Controllers, Drivers Seat, Dropbox, Data Integration Services, EastNet, Employee Learning Platform, EMS/GT Events, Enterprise Backup, Enterprise Data Warehouse, Enterprise Storage, Firewall, GitHub Enterprise, Global ECS, GT Reports, GTED, Jira, Kaltura Video Service, Endpoint Software Deployment and Update Tool, LAWN, LITE Reporting Portal, Log Management Services, Mailing Lists, Meal Plan, Microsoft Exchange Calendar, Microsoft Exchange Online, Microsoft OneDrive, Microsoft SharePoint, Microsoft Teams, My Appointment, My Housing, MyCloud, Network Access Control, Network Self-Service Applications, OMSCS, OneUSG Connect, PACE, Passport, Plesk, Presentation Computer, Print & Copy, Reporting and Analytics Tools, Resnet, Room Details and Schedule, Salesforce, Centralized Endpoint Management, Service Desk, ServiceNow-ITSM, ServiceNow-Security Operations, SingleSignOn (SSO), Sites@GT, Software Distribution, SolarWinds, System Services, T-Square, Techworks, Telecommunications, VEEAM, Virtual Server Hosting, VLAB, VPN, WAF - Web Application Firewall, Wireless Presentation System, Workday, Zoom, Office 365, Enrollment Management, Sunapsis, BuzzAPI

Description

WHAT IS HAPPENING? The Collaboration and Productivity Solutions team in OIT will be enhancing Georgia Tech's email security by updating email domains from their current SPF configuration of "SoftFail" to "Fail". The domains that will be updated are: • edm.gatech.edu • erp.gatech.edu • itg.gatech.edu • pace.gatech.edu • resnet.gatech.edu • services.gatech.edu • vtc.gatech.edu • mail.gatech.edu • oit.gatech.edu • auxtech.gatech.edu • is.gatech.edu • ts.gatech.edu • itsea.gatech.edu • security.gatech.edu • tag.gatech.edu • wrecktechs.gatech.edu Changing the SPF record from "SoftFail" to "Fail" has the potential to impact email deliverability. Email sent from unauthorized mail servers may be marked as junk by the receiving mail server or rejected outright. (More information on SPF is available in ServiceNow: https://gatech.service-now.com/home?id=kb_article_view&sysparm_article=KB0025892.) WHO IS AFFECTED BY THIS CHANGE? Only email sent from the domains listed above will be impacted if the sending service has not been properly authenticated. The Collab team has reviewed our DMARC reports since the previous change from "Neutral" to "SoftFail" to ensure that all legitimate servers and 3rd party mail senders have been properly configured. WHAT DO YOU NEED TO DO AFTER THIS CHANGE? Please report any unexpected email bounces for the domains listed above to the Collaboration and Productivity team at collab-support@gatech.gatech.edu. WHAT IF YOU HAVE QUESTIONS? If there are any questions, please contact the Collaboration and Productivity Solutions team at collab-support@gatech.edu.
Welcome to Georgia Tech's IT Service Status Page Don't see your issue posted here? Let us know! Administrative Services Center Email: support@oit.gatech.edu Phone: 404-385-1111 Location: Atlanta Campus, Clough Building Room 215