Flowroute has served Lime Daley for the last 15 years and they have done well for the most part, but every once in a while they make really bad policy decisions.  Last time they almost made me switch to a new provider was a silly decision to charge all customers a monthly fee for 911 access even if they couldn't actually call 911 due it being an incoming-only line or not even registered.  They insisted that they couldn't change that policy, so I cancelled a bunch of numbers that I had been holding onto because they had a good area and region code.  A month after I did that, they must have gotten yelled by lots of customers, because they reversed their position.

Last week, they made another change, disabling all outgoing SMS messages for all non-commercial customers, stating that all of their DIDs are commercial.  And since many of my customers are not commercial, they can't register for the 10DLC system that Flowroute is subjecting all of their customers to (including the increased fees).  So, I'm now in the market for a new SIP provider.

Until I can switch, all outgoing SMS messages won't be able to be sent.  I'm sort of hopeful that Flowroute might change their mind because it will be a lot of work to switch, and I've mostly been happy with their service (though we do have occasional issues in Hillsboro, NH when they switchover to cheaper routes, so hopefully that will go away with a new provider.  I'll be interviewing providers over the coming week or so. 

We are moving all email accounts to a new server which will improve reliability, security, and features.

We are hoping that customers won't notice any changes, but it is possible that there will be some connection issues, particularly if you have an old email client; it might need to be upgraded to connect to the new SSL certificates and ciphers.

Contact us if you have any problems in the coming weeks.

The servers have redundant A/B power connections, but there was a circuit breaker problem at the utility or something that somehow took out both power legs and didn't let the generator startup either.  Power went down at 3AM, and came back up at 4ish, but there are still some lingering affects.

Update: 06:52AM: Everything seems to be up and running at this point.  Let us know if you are still having any problems.

We lost a second drive in clementine today, which will affect some users and all phone service.  We are working on migrating the information to a new server and are hoping to have everything back up shortly.

Update: All users and phone service was moved to mandarin, and everything is back up functioning normally.

All of the servers need to be moved this weekend (the move has been in the works for the last three or four years...) and so there will be some downtime for each of the servers, around 15 minutes starting at 5:30AM, and finishing up by 7:30AM on Sunday morning.

Hopefully, this time frame will be as least intrusive as possible for you and your visitors/customers.

UPDATE: 2/21/2016 7:48AM: Everything went as well as it could have, and most of the servers had even less downtime than expected.  We took advantage of the downtime and ran all of the pending upgrades at the same time.  Let us know if you have any problems, but we've looked over log files and websites, and everything looks like it is working well.

We're increasing the security on orange to get it up to (mostly) modern standards.  The fingerprint for the RSA key is:

SHA256:mOIrhJUd+tgKjvMD21YvZaL3Ew37lqIubNZj0qz6Gjc

RSA:2048:4d:b4:d8:0d:0f:74:5f:68:2d:87:b7:df:fb:cb:80:6f

Sorry for the inconvenience, but is better to be inconvenienced than insecure...

Lime Daley experienced an unspecified hardware failure in a hard drive this morning, and caused some sites to be down for a number of hours.  The hardware is being worked on at the moment, and most customers are fully back up.  Some customers are still missing data, and that data will hopefully be automatically restored over the next couple of hours.

Lime Daley has updated all of our SSL certificates due to the recently discovered bug in the widely used OpenSSL libraries.

The vulnerability affected our email and web services.  Unfortunately, there isn't any way to detect whether an attacker has made any use of this vulnerability or not.  To be safe, you should update any passwords you have.

This vulnerability does not apply to those customers on "orange", as that server was using a version of the software that was not affected by this bug.

Additionally, access via sftp and ssh were also not affected.

Any services not protected by SSL (like web pages that don't have an https:// at the front, but just http://)  are also not affected, because non-ssl services are eavesdroppable by default any way.

For more information, you can read about it at heartbleed.com.  You can also test any services you would like at filippo.io.

A hard drive failed in clementine, and the plan is to replace the drive today, provided that USPS delivered it to the right place.  I'm hoping to replace it without taking the machine down, but we might need to power down the machine for a little while to avoid losing any data during the replacement.

I'll update this page as any new information happens.

10:30AM Update: The drive has been replaced, and the machine is back up and running.  Let me know if you see any problems, but everything looks good on my end.  The server might be a little slow today as it copies the data to the new drive.

Since the servers are located on the east coast, we are potentially affected by Hurricane Sandy.  But, the backup power systems were tested this morning, generators have their fuel tanks topped up, and we are not expecting any problems with the service.  Stay safe at home checking your email...  :)