Georgia Tech IT
Active Incident

Updated a few seconds ago

Barracuda WAF issue affecting Plesk HostingDegraded Performance

Incident Status

Degraded Performance

Components

Web Hosting

Services

Plesk, WAF - Web Application Firewall


April 24, 2024 8:47AM EDT
[Monitoring] OIT is continuing to monitor the situation, and there have been no significant updates yet. Support teams are still working with the vendor to get a more permanent fix to the issue.

April 16, 2024 3:35PM EDT
[Monitoring] While the workaround to address the WAF issue on the the Plesk Webhosting complex on 4/5 did restore service, it unfortunately also is causing an IP forwarding issue which is affecting the functionality of some of the sites. Support is continuing to work with the vendor to restore the WAF to service so that the webhosting complex can be brought back to its original state.

April 12, 2024 2:51PM EDT
[Monitoring] The workaround that was put in to restore connectivity to plesk hosted websites continues to be working. However, a permanent solution to the underlying causing is still being investigated and worked on with the vendor. Once a working solution is found, tested, and implemented, this issue will be cleared . Until then, we will continue to run off the workaround, and support is continuously monitoring the situation for any new issues that could spring up from the workaround.

April 6, 2024 12:34AM EDT
[Monitoring] The workaround was successful, and sites should be accessible once again. Support is continuing to work with the vendor to address the underlying issue for a more permanent fix.

April 5, 2024 11:33PM EDT
[Identified] Support is continuing to work with the vendor to try and fix the issue with the WAF. However, network team is currently getting ready to implement a workaround to restore connectivity to the affected sites as a temporary measure until a fix for the WAF issue can be found. Once the workaround is implemented, approximately 30 minutes of down time on the hosted sites is expected.

April 5, 2024 1:51PM EDT
[Identified] Staff are still working with the vendor to resolve the WAF issue. If students need to access Canvas, they can reach it by going to https://gatech.instructure.com/

April 5, 2024 11:06AM EDT
[Identified] The vendor's patch was unsuccessful in resolving the issue and will rollback their patch. Efforts are being made to prevent any downtime, but it is possible that there may be a brief outage of up to 10 minutes. Support is continuing to work with the vendor to troubleshoot the problem.

April 5, 2024 9:23AM EDT
[Identified] The vendor is applying a patch to address the communications issues with the Web Application Firewall. Efforts are being made to prevent any downtime, but it is possible that there may be a brief outage of up to 10 minutes.

April 5, 2024 8:33AM EDT
[Identified] Staff have discovered a communications issue between the Plesk hosts and the WAF (web application firewall) that is causing sites hosted on Plesk to be slow or unreachable. Staff are continuing to work with the vendor to resolve this issue. Sites affected include but are not limited to: buzzport.gatech.edu canvas.gatech.edu crc.gatech.edu ctl.gatech.edu dining.gatech.edu diversity.gatech.edu finance.gatech.edu honors.gatech.edu hosting.gatech.edu housing.gatech.edu ipat.gatech.edu

April 5, 2024 8:00AM EDT
[Investigating] Several sites hosted on the webplesk are unreachable due to issues with the WAF (web application firewall) including but not limited to : buzzport.gatech.edu canvas.gatech.edu crc.gatech.edu ctl.gatech.edu dining.gatech.edu diversity,gatech.edu housing.gatech.edu ipat.gatech.edu Staff are continuing to investigate with the vendor to resolve this issue.

April 4, 2024 4:58PM EDT
[Investigating] Several sites hosted on the webplesk are unreachable due to issues with the WAF (web application firewall) including but limited to : canvas.gatech.edu crc.gatech.edu ctl.gatech.edu dining.gatech.edu diversity,gatech.edu housing.gatech.edu ipat.gatech.edu

April 4, 2024 2:29PM EDT
[Monitoring] Service is currently restored and support is monitoring the situation. If you are continuing to experience issues, or if you have questions or concerns, please file a ticket via services.gatech.edu or contact the Administrative Services Center at 404/385-1111.

April 4, 2024 11:50AM EDT
[Identified] Related to the previous issue, web-plesk53 is now serving incorrect SSL certificates to sites hosted on the server. Support is working to resolve the issue.

April 4, 2024 9:49AM EDT
[Monitoring] Several sites hosted off of web-plesk53 were unreachable due to issues with the WAF (web application firewall) including: -Housing.gatech.edu, textbooks.math.gatech.edu, -undergradresearch.gatech.edu, -faset.gatech.edu, -transitionprograms.gatech.edu, -policylibrary.gatech.edu, -policies.gatech.edu, -alan.ece.gatech.edu -gtcad.gatech.edu -hasler.ece.gatech.edu Service is currently restored and support is monitoring the situation.

Academic Services

Operational

Campus Services

Operational

Campus Services - ITG

Operational

Campus Audio Visual Services

Operational

Email and Calendaring

Operational

Identity

Operational

Network

Operational

Web Hosting

Operational

Enterprise Data Services

Operational

IT Service Management

Operational

VideoConferencing

Operational

GTPE

Operational

Cloud Storage

Operational

External Services

AWS CloudFront

AWS EC2

Azure

Box

DocuSign

Dropbox

Duo

Honorlock

Instructure (Canvas)

Kaltura

LastPass

Let's Encrypt

MS Office365

MuleSoft

Salesforce

Sectigo

Zoom

Scheduled Maintenance

PACE Maintenance Period (May 07 – May 10, 2024) Planned Maintenance

Schedule

May 7, 2024 6:00AM - May 10, 2024 11:59PM EDT

Components

Academic Services

Services

PACE

Description

ITEMS REQUIRING USER ACTION: [all] During the maintenance period, the PACE team will rename all POSIX user groups so that names will start with the “pace-” prefix. This will NOT affect numerical GIDs, but if your scripts or workflows rely on group names, they will need to be updated. If you don’t use POSIX user group names in your scripts or workflows, no action is required on your part. This is a step towards tighter integration of PACE systems with central IAM tools, which will lead to improvements across the board in the PACE user experience. NOTE: This item was originally planned for January, but was delayed to avoid integration issues with IAM services, which have now been resolved. [ICE] Migrate to the RHEL 9.3 operating system – if you need access to ICE for any summer courses, please let us know! Note – This WILL create new ssh host-keys on ICE login nodes – so please expect a message that “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!” when you (or your students) next access ICE after maintenance. ITEMS NOT REQUIRING USER ACTION: [datacenter] Databank maintenance: replace all components of cold loop water pump that had issues a couple of maintenance periods ago. [Hive] Upgrade the underlying GPFS filesystem to version 5.1 in preparation for RHEL9 [datacenter] Repairs to one InfiniBand switch and two DDN storage controllers with degraded BBUs (Battery Backup Unit) [datacenter] Upgrade storage controller firmware for DDN appliances to SFA 12.4.0. WHO IS AFFECTED? All users across all PACE clusters. WHO SHOULD YOU CONTACT FOR QUESTIONS? Please contact PACE at pace-support@oit.gatech.edu with questions or concerns.
Email service maintenance - SPF update to SoftFail for OIT email domainsPlanned Maintenance

Schedule

May 14, 2024 7:00AM - 9:00AM EDT

Components

Email and Calendaring

Services

Microsoft Exchange Online

Description

WHAT IS HAPPENING? The Collaboration and Productivity Solutions team in OIT will be enhancing Georgia Tech's email security by updating email domains from their current SPF (Sender Policy Framework) configuration of "Neutral" to "SoftFail". The domains that will be updated are (CHG0033114): • edm.gatech.edu • erp.gatech.edu • itg.gatech.edu • pace.gatech.edu • resnet.gatech.edu • services.gatech.edu • vtc.gatech.edu • mail.gatech.edu • oit.gatech.edu SPF (Sender Policy Framework) is a record in DNS that lists which email systems Georgia Tech authorizes to send email as GT email domains. Changing the SPF record from "Neutral" to "SoftFail" has the potential to impact email deliverability if an email sender is not listed in the SPF record or DKIM signing mail. Email sent from unauthorized mail servers (servers not listed in the SPF record or DKIM signing) may be marked as junk by the receiving mail server and would result in the email being delivered to the recipients Junk Email/Spam folder. Some email systems outside Georgia Tech may choose to reject the mail. (More information on SPF is available in ServiceNow: https://gatech.service-now.com/home?id=kb_article_view&sysparm_article=KB0025892.) WHO IS AFFECTED BY THIS CHANGE? Only email sent from the domains listed above will be impacted if the sending service has not been properly authenticated. The Collab team has reviewed our DMARC reports to ensure that all legitimate servers and 3rd party mail senders have been properly authenticated. DMARC reports currently indicate that no legitimate mail flow should be impacted by this change. WHAT DO YOU NEED TO DO AFTER THIS CHANGE? Please report any unexpected email bounces for the domains listed above to the Collaboration and Productivity team at collab-support@gatech.gatech.edu. WHAT IF YOU HAVE QUESTIONS? If there are any questions, please contact the Collaboration and Productivity Solutions team at collab-support@gatech.edu.
Email service maintenance - SPF update to Fail for OIT email domainsPlanned Maintenance

Schedule

May 27, 2024 7:00AM - 9:00AM EDT

Components

Academic Services, Campus Services, Campus Services - ITG, Campus Audio Visual Services, Email and Calendaring, Identity, Network, Web Hosting, Enterprise Data Services, IT Service Management, VideoConferencing, GTPE, Cloud Storage

Services

Active Directory, Top Level Web - AWS, Azure Active Directory, Banner and Oscar, BigIP and GTM, Box, BuzzCard - 3rd Party Retail Merchants, BuzzCard - Card Production, BuzzCard - Copier Readers, BuzzCard - Laundry Readers, BuzzCard - Online Deposits, BuzzCard - Pay for Print, Buzzport, Cable TV (GTCN), Campus Network, Canvas, Cloudbolt, Contract Information System, Centralized Endpoint Backups, ctr.gatech.edu, DegreeWorks, Deltek Costpoint, DHCP, Digital Room Indicator, DNS, Door Controllers, Drivers Seat, Dropbox, Data Integration Services, EastNet, Employee Learning Platform, EMS/GT Events, Enterprise Backup, Enterprise Data Warehouse, Enterprise Storage, Firewall, GitHub Enterprise, Global ECS, GT Reports, GTED, Jira, Kaltura Video Service, Endpoint Software Deployment and Update Tool, LAWN, LITE Reporting Portal, Log Management Services, Mailing Lists, Meal Plan, Microsoft Exchange Calendar, Microsoft Exchange Online, Microsoft OneDrive, Microsoft SharePoint, Microsoft Teams, My Appointment, My Housing, MyCloud, Network Access Control, Network Self-Service Applications, OMSCS, OneUSG Connect, PACE, Passport, Plesk, Presentation Computer, Print & Copy, Reporting and Analytics Tools, Resnet, Room Details and Schedule, Salesforce, Centralized Endpoint Management, Service Desk, ServiceNow-ITSM, ServiceNow-Security Operations, SingleSignOn (SSO), Sites@GT, Software Distribution, SolarWinds, System Services, T-Square, Techworks, Telecommunications, VEEAM, Virtual Server Hosting, VLAB, VPN, WAF - Web Application Firewall, Wireless Presentation System, Workday, Zoom, Office 365, Enrollment Management, Sunapsis, BuzzAPI

Description

WHAT IS HAPPENING? The Collaboration and Productivity Solutions team in OIT will be enhancing Georgia Tech's email security by updating email domains from their current SPF configuration of "SoftFail" to "Fail". The domains that will be updated are: • edm.gatech.edu • erp.gatech.edu • itg.gatech.edu • pace.gatech.edu • resnet.gatech.edu • services.gatech.edu • vtc.gatech.edu • mail.gatech.edu • oit.gatech.edu • auxtech.gatech.edu • is.gatech.edu • ts.gatech.edu • itsea.gatech.edu • security.gatech.edu • tag.gatech.edu • wrecktechs.gatech.edu Changing the SPF record from "SoftFail" to "Fail" has the potential to impact email deliverability. Email sent from unauthorized mail servers may be marked as junk by the receiving mail server or rejected outright. (More information on SPF is available in ServiceNow: https://gatech.service-now.com/home?id=kb_article_view&sysparm_article=KB0025892.) WHO IS AFFECTED BY THIS CHANGE? Only email sent from the domains listed above will be impacted if the sending service has not been properly authenticated. The Collab team has reviewed our DMARC reports since the previous change from "Neutral" to "SoftFail" to ensure that all legitimate servers and 3rd party mail senders have been properly configured. WHAT DO YOU NEED TO DO AFTER THIS CHANGE? Please report any unexpected email bounces for the domains listed above to the Collaboration and Productivity team at collab-support@gatech.gatech.edu. WHAT IF YOU HAVE QUESTIONS? If there are any questions, please contact the Collaboration and Productivity Solutions team at collab-support@gatech.edu.
Welcome to Georgia Tech's IT Service Status Page Don't see your issue posted here? Let us know! Administrative Services Center Email: support@oit.gatech.edu Phone: 404-385-1111 Location: Atlanta Campus, Clough Building Room 215