Camayak Blog

Camayak is a content production tool for newsrooms.
Empower editors. Improve communication. Create better content.

Service Updates

  1. Temporary downtime this weekend

    ,

    Tonight we’re going to be performing system maintenance, which we expect to take one hour. You won’t be able to access Camayak during these periods so please make sure you’ve saved your work before we get started.

    Camayak may be accessible more quickly than we anticipate, so keep an eye on our Twitter account for updates on when you can resume your work.

    Depending on your location, here’s when Camayak will be unavailable:

    August 5/6th 2017:

    • Central EU: 3 a.m. – 4 a.m., Tomorrow
    • UK: 2 a.m. – 3 a.m., Tomorrow
    • Eastern US: 9 p.m. – 10 p.m., Today
    • Central US: 8 p.m. – 9 p.m., Today
    • Mountain US: 7 p.m. – 8 p.m., Today
    • Pacific US: 6 p.m. – 7 p.m., Today
    • Hawai’i: 2 p.m. – 12 a.m., Today

    We’re sorry for any inconvenience this causes and look forward to updating our Twitter account when we’re finished with our work.

  2. Camayak scheduled downtime on July 4th

    ,

    On July 4th, we’re going to be performing system maintenance, which we expect to take one hour. You won’t be able to access Camayak during these periods so please make sure you’ve saved your work before we get started.

    Camayak may be accessible more quickly than we anticipate, so keep an eye on our Twitter account for updates on when you can resume your work.

    Depending on your location, here’s when Camayak will be unavailable:

    July 4th 2017:

    • Central EU: 11 a.m. – 12 p.m
    • UK: 10 a.m. – 11 a.m
    • Eastern US: 5 a.m. – 6 a.m
    • Central US: 4 a.m. – 5 a.m
    • Mountain US: 3 a.m. – 4 a.m
    • Pacific US: 2 a.m. – 3 a.m
    • Hawai’i: July 3rd 11 p.m. – 12 a.m., July 4th

    We’re sorry for any inconvenience this causes and look forward to updating our Twitter account when we’re finished with our work.

  3. Camayak scheduled downtime on June 25

    ,

    On June 25th, we’re going to be performing system maintenance, which we expect to take two hours. You won’t be able to access Camayak during these periods so please make sure you’ve saved your work before we get started.

    Camayak may be accessible more quickly than we anticipate, so keep an eye on our Twitter account for updates on when you can resume your work.

    Depending on your location, here’s when Camayak will be unavailable:

    June 25th 2017:

    • Central EU: 11 a.m. – 1 p.m
    • UK: 10 a.m. – 12 p.m
    • Eastern US: 5 a.m. – 7 a.m
    • Central US: 4 a.m. – 6 a.m
    • Mountain US: 3 a.m. – 5 a.m
    • Pacific US: 2 a.m. – 4 a.m
    • Hawai’i: Saturday 24th 11 p.m. – 1 a.m., Sunday 25th

    We’re sorry for any inconvenience this causes and look forward to updating our Twitter account when we’re finished with our work.

  4. Camayak scheduled downtime on June 6

    ,

    On June 6th, we’re going to be performing system maintenance, which we expect to take up to three hours. During this time the Camayak software will be available, but the Camayak product website will be unavailable.

    If you usually login via camayak.com, during this window you will need to login directly via my.camayak.com instead.

    June 6th 2017:

    • Central EU: 2 p.m. – 5 p.m
    • UK: 1 p.m. – 4 p.m
    • Eastern US: 8 a.m. – 11 a.m
    • Central US: 7 a.m. – 10 a.m
    • Mountain US: 6 a.m. – 9 a.m
    • Pacific US: 5 a.m. – 8 a.m
    • Hawai’i: 2 a.m. – 5 a.m

    We’re sorry for any inconvenience this causes.

  5. Camayak scheduled downtime this Sunday morning

    ,

    This Sunday, December 13th, we’re going to be performing system maintenance, which we expect to take two hours. You won’t be able to access Camayak during these periods so please make sure you’ve saved your work before we get started.

    Camayak may be accessible more quickly than we anticipate, so keep an eye on our Twitter account for updates on when you can resume your work.

    Depending on your location, here’s when Camayak will be unavailable:

    Sunday 13th, December 2015:

    • Central EU: Sunday, 11 a.m. – 1 p.m.
    • UK: Sunday, 10 a.m. – 12 p.m.
    • Eastern US: Sunday, 5 a.m. – 7 a.m.
    • Central US: Sunday, 4 a.m. – 6 a.m.
    • Mountain US: Sunday, 3 a.m. – 5 a.m.
    • Pacific US: Sunday, 2 a.m. – 4 a.m.
    • Hawai’i: Saturday 12th 11 p.m. – 1 a.m., Sunday 13th

    We’re sorry for any inconvenience this causes and look forward to updating our Twitter account when we’re finished with our work.

  6. Camayak scheduled downtime on Monday and Wednesday this week

    ,

    Tomorrow and Wednesday October 21st, we’re going to be performing system updates. You won’t be able to access Camayak during these periods so please make sure you’ve saved your work before we get started.

    Camayak may be accessible more quickly than we anticipate, so keep an eye on our Twitter account for updates on when you can resume your work.

    Depending on your location, here’s when Camayak will be unavailable:

    Monday October 19th 2015:

    • Central EU: Monday, 1 p.m. – 3 p.m.
    • UK: Monday, 12 p.m. – 1 p.m.
    • Eastern US: Monday, 7 a.m. – 9 a.m.
    • Central US: Monday, 6 a.m. – 8 a.m.
    • Mountain US: Monday, 5 a.m. – 7 a.m.
    • Pacific US: Monday, 4 a.m. – 6 a.m.
    • Hawai’i: Monday, 1 a.m. – 3 a.m.

    Wednesday October 21st 2015:

    • Central EU: Wednesday, 8:45 a.m. – 11:00 a.m.
    • UK: Wednesday, 7:45 a.m. – 10:00 a.m.
    • Eastern US: Wednesday, 2:45 a.m. – 5:00 a.m.
    • Central US: Wednesday, 1:45 a.m. – 4:00 a.m.
    • Mountain US: Wednesday, 12:45 a.m. – 3:00 a.m.
    • Pacific US: Tuesday, 11:45 p.m. – 2:00 a.m.
    • Hawai’i: Tuesday, 8:45 p.m. – 12:00 a.m.

    We’re sorry for any inconvenience this causes and look forward to updating our Twitter account when we’re finished with our updates.

  7. Scheduled downtime

    ,

    Tomorrow we’re going to be performing system updates. You won’t be able to access Camayak during this period so please make sure you’ve saved your work before we get started.

    If you’re based in Europe you’re most likely to be affected by this, so it’s important to mark these times on your calendar. Camayak may be accessible more quickly than we anticipate, so keep an eye on our Twitter account for updates on when you can resume your work.

    Depending on your location, here’s when Camayak will be unavailable:

    • UK: Wednesday, 10 a.m. – 11 a.m.
    • Central EU: Wednesday, 11 a.m. – 12 p.m.
    • Eastern US: Wednesday, 5 a.m. – 6 a.m.
    • Central US: Wednesday, 4 a.m. – 5 a.m.
    • Mountain US: Wednesday, 3 a.m. – 4 a.m.
    • Pacific US: Wednesday, 2 a.m. – 3 a.m.
    • Hawai’i: Tuesday, 11 p.m. – 12 a.m.

    We’re sorry for any inconvenience this causes and look forward to updating our Twitter account when we’re finished with our updates.

  8. Tomorrow Evening, Use my.camayak.com to Access Your Newsroom

    ,

    Starting on Sunday evening, 30th March, you’ll need to go directly to my.camayak.com to login to your account, as www.camayak.com may temporarily be unavailable.

    Please remember this between:

    Eastern Time: 12am and 9am (Monday)

    Central Time: 11pm and 8am (Monday)

    Mountain Time: 10pm and 7am (Monday)

    Pacific Time: 9pm and 6am (Monday)

    Hawai’i Time: 7pm and 4am (Monday)

    P.S. If you bookmark my.camayak.com you can save going to www.camayak.com, every time you want to access your newsroom.

  9. Scheduled downtime: Wednesday & Thursday this week

    ,

    Tomorrow we’re going to be performing system updates. You won’t be able to access Camayak during this period so please make sure you’ve saved your work before we get started. We’re going to repeat the same thing on Thursday morning so the same will apply, then.

    If you’re based in Europe you’re most likely to be affected by this, so it’s important to mark these times on your calendar. Camayak may be accessible more quickly than we anticipate, so keep an eye on our Twitter account for updates on when you can resume your work.

    Depending on your location, here’s when Camayak will be unavailable:

    Central EU: Wednesday & Thursday, 1 p.m. – 3 p.m.

    UK: Wednesday & Thursday, 12 p.m. – 2 p.m.

    Eastern US: Wednesday & Thursday, 7 a.m. – 9 a.m.

    Central US: Wednesday & Thursday, 6 a.m. – 8 a.m.

    Mountain US: Wednesday & Thursday, 5 a.m. – 7 a.m.

    Pacific US: Wednesday & Thursday, 4 a.m. – 6 a.m.

    Hawai’i: Wednesday & Thursday, 2 a.m. – 4 a.m.

    We’re sorry for any inconvenience this causes and look forward to updating our Twitter account when we’re finished with our updates.

    Update:

    Thursday’s scheduled downtime has been cancelled.

  10. Root Cause Analysis for Camayak Outage of February 3rd/4th

    ,

    Overview

    Camayak customers experienced three service outages though the evening hours of February 3rd (US Time Zones)/Early Morning February 4th UTC.

    The outages spanned the following times:

    Outage 1 (34 minutes):

    – UTC From 02/04/2015 03:39 to 02/04/2015 04:13

    – CST From 02/03/2015 21:39 to 02/03/2015 22:13

    Outage 2 (12 minutes):

    – UTC From 02/04/2015 04:37 to 02/04/2015 04:49

    – CST From 02/03/2015 22:37 to 02/03/2015 22:49

    Outage 3 (40 minutes):

    – UTC From 02/04/2015 04:54 to 02/04/2015 05:34

    – CST From 02/03/2015 22:54 to 02/03/2015 23:34

    In the times between the outages, service may have been slow as customers tried to reconnect.

    Part 1: The service outage

    A: Cause

    We have determined that the service outages were caused by the following circumstances:

    1) A customer inadvertently created a large number of publishing destinations pointing to non-existent or misconfigured WordPress instances.

    2) When the customer went to the manage publishing destinations page, Camayak attempted to validate the publishing destinations. Due to the specific circumstances of those WordPress instances, the validation code ended up waiting a long time for validation before timing out.

    3) The validation attempts tied up application server threads, preventing them from being used to service other customer’s requests

    4) Our load balancer, seeing that the applications servers were not responding to the validation requests in a timely fashion re-attempted those requests to other application servers, causing application server threads on those servers to be tied up waiting for validation.

    5) This process continued until all application server threads were tied up waiting for validation.

    6) Eventually, the validation requests timed out fully, and the application server threads were available to process other requests.

    Steps 2-6 of this process happened three times, causing the three outages.

    B: Remediation

    1) We have, as of February 4th 16:45 UTC (10:45 CST) released a fix to our application server software to ensure that reasonable timeouts on the validation attempts. The validation timeout is well below the load balancer timeout, to ensure the cascade effect in (4) does not happen.

    2) We have, as an additional measure, increased our application server capacity.

    Part 2: The support outage

    A: Cause

    The Camayak support team was unable to respond to support queries from customers, which began at UTC 04:01, until UTC 07:29. The reasons for this are detailed here.

    B: Remediation

    We have, as of February 4th 14:45 UTC (08:45 CST) set up a notifications system with extra features to alert and inform off duty support personnel of service outages. This is an addition to our existing notifications framework for other critical and non-critical alerts.