Oct 21, 2021 11:54AM - Amex outage - Resolved
Issue: the American Express credit card processing network is experiencing an outage that is affecting clients trying to charge Amex cards through our integrated merchant services processors. See status.bluefin.com for the most up-to-date updates on this issue.
Aug 12, 2021 8:26-11:06AM - EPCS routing problems - Resolved
Issue: problems with routing controlled substances. Our eRx two-factor verification partner encountered server issues with routing controlled substances but they have confirm sends are now processing successfully.
July 21, 2021 9:26-10:40AM - SMS sending issues - Resolved
Issue: outbound SMS messages failing to send. Failure was addressed by the carrier.
June 1, 2021 7:12-10:36AM - Zoom phone issues - Resolved
Zoom identified the issue causing latency with Zoom Phone service when accessing the web portal and SMS messaging. They will continue to provide updates https://status.zoom.us/ but currently seems to be mostly resolved.
May 10, 2021 7:15-8:10AM Pacific - Latency Issues - Resolved
Issue: latency/ slowness. Reports of latency this morning appear to be resolved after a small-scale DOS attack on our firewall this morning. Our host migrated our primary firewall and implemented some geographic blocking rules which appeared to resolve the issue.
April 13, 2021 5:30AM - 2:48PM - Latency - Resolved
Issue: Latency/ slowness affecting ~20% of users. Our host successfully repaired the root cause of the latency that around 20% of our users were experiencing today. A physical interface issue on one of the data-center's network switches was causing an unacceptable level of packet loss on outbound traffic for nodes that were connected through the faulty switch. This interface has been removed from the network and replaced and traffic speeds immediately returned to normal. We're waiting on a full post mortem from our host's network team, but in the interim load times should be dramatically faster.
March 25, 2021 9:21-9:54AM - Latency - Resolved
Issue: low-level latency/ slowness affecting the majority of clients. Most users would not have noticed this, but some services were moving slower than usual. Our host rolled out an update to address firewall issues that resolved the problem.
March 22, 2021 - performance issues related to Active Campaign integration - Resolved
Customers who have Active Campaign integrated into their Cerbo accounts were reporting performance issues. We disabled the integrations until Active Campaign's performance recovered. Then re-enabled the syncs.
Jan 20, 2021 8:00-8:12AM - Inbound fax errors - Resolved
Our Fax carrier's API was returning errors, but it appears that they've resolved the issue. If you saw red error messages on some fax-related screens (inbound fax queue, sending a new fax, and loading the Sent document tab), those messages should disappear when you refresh the screen.
Dec 16, 2020, 9:05-11:00AM - Latency - Resolved
Some builds were experiencing intermittent slow performance. The issue was caused by significantly higher than normal traffic on our central servers, which caused a slowdown at the firewall. We will be upgrading the firewall after hours tonight to avoid a recurrence of the issue. Slowdown issues appear to have resolved as of about 11am Pacific, but we will continue to monitor this issue throughout the day. In the meantime, anyone who runs into slowness should find that the issue resolves itself within a few minutes.
July 28, 2020, 10:10-11:35AM - Outbound email issues - Resolved
Primary delivery service (Sendgrid) was experiencing API issues that were causing outbound emails to be rejected a small percentage of the time. We temporarily rolled over to an alternate mail service. And rolled back to the primary when the issue was resolved.
Jun 12, 2020 7:44-8:00AM - Brief EHR outages for ~15% of builds - Resolved
A server node rebooted unexpectedly, causing outages of between 2 and 15 minutes. This is the same node that experienced an outage last week, and the root cause is still unclear. So we are migrating all builds off of the problematic node.
Jun 4, 2020 2:13-2:35PM PST - Brief EHR outages for ~15% of builds - Resolved
Handful of outages in one server node. Cause was not able to be determined at the time. Actions taken to prevent recurrence: several builds were migrated off of this node to free up additional space/ resources, and additional monitoring was implemented.
Apr 21, 2020 8:30AM-10AM - Major internet carrier issues (not Cerbo-related) - Resolved
One of the major cross-country data lines is experiencing an outage, which is causing speed and connection issues for many connecting from western states - Colorado in particular (that is where the outage is located), but also clients whose internet connection routes through Colorado.Our hosting company dropped their Level 3 routes to force connection into the Cogent network and this appears to have resolved the issue by around 10AM PST.
Apr 4, 2020 11:40-12:08PM - EHR outage affecting ~10% of builds - Resolved
The same node shut down again, impacting ~10% of our clinics. Engineers determined that it was a faulty DIMM chip on the affected node and did an emergency hardware swap to replace the system RAM. All services should be restored and engineers are monitoring the host.
Feb 3, 2020 4:09-4:41PM - EHR outage affecting ~10% of builds - Resolved
Cloud server node went down, causing outage for some customers.
Jan 6, 2020 - Payment processing issues - Resolved
Bluefin servers are down resulting in failed credit card payment processing for some customers.
Nov 4th, 2019 - Performance issues affecting less than 10% of builds - Resolved
High CPU load on one node. Performance issues resolved, migration scheduled for after hours 10/4/2019 for permanent resolution.
