St. Paul to begin restoring online systems this week

The city of St. Paul expects to begin putting its systems back online this week, after a cyberattack forced it to shut down its network for more than two weeks.

St. Paul officials said Saturday night that the attack was a ransomware attack, adding that they’ve refused to pay the ransom and haven’t found evidence of data theft, But it's been disruptive and costly.

Since July 25, residents haven't been able to pay utility bills online or access free city WiFi, among other inconveniences. Emergency services and 911 have remained operational.

St. Paul Mayor Melvin Carter joined MPR News guest host Emily Bright on Morning Edition Monday to talk more about next steps for security and the community.

The following has been lightly edited for clarity. Listen to the full conversation by clicking the player button above.

How did you first learn about the attack?

We've spent a lot of effort over the last couple of years boosting our cybersecurity profile as a city. We created a full-time chief information security officer position. She's been incredible, boosting multi-factor authentication and the internal spam efforts just to try to make sure that we're boosting our profiles.

On July 25, our systems — as they're supposed to do — flagged some suspicious activity in one of our servers and our staff went to work really hard right away segmenting our systems off, backing up data and those types of things, and trying to identify where the threat was coming from, so that we can make sure that we could contain it.

The service outage has lasted more than two weeks, and communication about progress had been scarce. What do you say to accusations saying you moved too slowly?

Well, I imagine some of those folks may have never dealt with a cyberattack of this magnitude. We're also in between an FBI-run criminal investigation and a group of hackers, somebody out there who's watching closely everything that we say. So, of course, we don't want to say anything that would give them an opportunity to double down on what they're trying to do to our systems.

I've been really proud of the way — and frankly, the pace — at which city employees have done this. With 3,500 city employees, our staff, led by our Chief Information Officer Jaime Wascalus, have, in the last two weeks, pored through every server that we have, every device that we have. Right now, we're going through this thing called Operation Secure St. Paul, in which, over the course of three days, we're manually resetting in person 3,500 city employees’ passwords. That's a pretty, actually blinding pace that our city staff has gone through.

I know there are some inconveniences along the way, but one of the things that we've wanted to make sure that residents understand is that every service is available. The city employees who respond to 911 calls, who take out our trash, who file building permits, are excited to do those things. They may not have some of the apps that they're used to working with, but we've continued to maintain the operational security. So our missions have been one, to do that forensic evaluation, and two, to maintain operational continuity, 911 services, trash collection, payroll — those types of things, frankly, have been higher priorities than getting our networks back up — and then three, moving to get our networks back up.

We're excited about this Operation Secure St. Paul. It's sort of the final step of a grand control-alt-delete of all of our city systems, and that'll put us in a position to start putting systems back online.

How soon will systems start coming back online?

Operation Secure St. Paul is a three-day effort that started early Sunday morning. And so they are working right now every half hour, bringing 180 city staff through to reset their passwords. And again, once these three days are complete, that'll be sort of the culmination of this grand reset that we've done on all of our systems, and we'll start bringing our kind of most critical systems back online by the end of this week.

In addition to getting help from law enforcement, you’re working with a private firm. How much is that firm costing taxpayers?

We don't know yet. One of the things that I've shared with folks is, you know, when there's a fire, we put out the fire, and we talk about water rates later. And so we've been responding to the emergency, the cost of this cyberattack will be significant. And, you know, we don't have all of that kind of put together yet. Obviously, we have to respond to the emergency. There's no other way around it, and not doing all the work that we're doing right now would be far more expensive than doing the work that we're doing right now. We'll have to put that together.

We've been having conversations with our city council so they understand that that's all coming. There has been an enormous amount of overtime as our staff has worked literally around the clock for the last couple of weeks to make sure that we put our city in the position that we're in right now.

What has the city learned from this incident, and what will St. Paul do to prevent such an attack from happening again?

The magnitude and the sophistication of cyberattacks have just blown up over the last, even, five years. We're seeing literally every government unit, every school, every hospital, you know, every institution has to be concerned and has to think about their kind of cybersecurity protocols.

We have put in place significant cybersecurity protocols, and one of the things that we're hearing from the experts right now is that, again, given the volume, given the magnitude, given the sophistication, that the expectation really shouldn't be at this point, that you can build a brick wall that nobody ever can get around or get through. The question is, how do you build the systems to identify them as soon as possible, as our systems did, and the people protocols to respond to them in a way that can put us in the type of position that we're in right now?

There's some real, incredible successes. There's obviously some things to button up. And one of the big things that we've decided to do is identify the systems, the cybersecurity systems that flagged it first. We're making sure that we've got that on all of our devices in the city. That's a part of that grand reset that I was talking about.

So figuring out how you can be nimble toward any future?

All of the above. Getting the enhanced cybersecurity software on all of those devices. Again, this grand reset of all of our passwords is going to help enormously. And here's the thing, whatever organization we all work at, I know that we've all probably had some experience sort of ignoring that refresh your password thing. We can't do that anymore. We have to be kind of vigilant where that's concerned.

And we're going to rely on the vigilance of all of our staff. You know, we have 3,500 teammates in our city, and we have to rely on the vigilance of every single one of them every single day to help keep those systems secure for all of us.