Camayak Blog

Camayak is a content production tool for newsrooms.
Empower editors. Improve communication. Create better content.

Service Updates

  1. We’re updating our Privacy Terms

    ,

    You’ve probably heard that from May 25th, people living in Europe will have an easier time understanding how their personal data is being handled by companies that ask for it. This is a summary of how Camayak is working to comply with the new regulations and a resource for people interested in their ramifications.

    Within GDPR (General Data Protection Regulation), companies that acquire personal data relating to individuals are reminded that:

    • They must process and store people’s data securely
    • They must be transparent about who that data passes on to and why
    • They must allow their customers to audit their fulfillment of their claims

    The new GDPR directives also have the welcome effect of incentivizing companies to offer you the same transparency due to European residents; no matter where you live. For example: even if you don’t support individual privacy (we do), it’s quite simply more efficient for small and mid-sized web-based software services like Camayak to assume that GDPR applies to all our customers than to try and keep an unregulated relationship with customers that don’t live in the European Union.

    But there’s more to complying with the new rules than giving your company’s terms and conditions a Spring clean. Helpfully there are much larger companies that operate in similar ways to Camayak which we’ve been able to learn from as they look to satisfy the new guidelines. Here are some of the highlights that we’ve appreciated so far:

    • Zendesk keep a really clear list of the people (nicknamed ‘subprocessors’) that they work with who may also be handling personal data from Zendesk’s customers. We’ve created something similar.
    • Google Ads invite customer audits and thoroughly demonstrate how they work to protect personal data from being breached.
    • Slack introduced a profile deletion tool. While allowing users to request that their profiles be deleted, neither Slack nor Camayak offers its users the ability to delete content that they’ve created in quite the same way. This has been a hot button topic for some of our clients in the past, as individuals have requested that some of their past work – or even content where they are the subject matter – be redacted from the public domain. Camayak’s default position remains that such decisions are down to the organization owners/editors.

    Joining many other companies, we’re updating our Privacy Policy and and Terms of Service to make our business goals and practices even more transparent, without collecting more personal data or changing the way we handle it. Here are some of the more substantial changes we’re making in order to adequately meet the expectations of the new legislation:

    • Children. To use Camayak you must now be 16 or older.
    • Your rights. We’ve made it clearer how you can exercise your data rights.
    • Data deletion. We’re sympathetic to the appropriate reasons for someone wanting to have certain personal data completely removed from Camayak.
    • Use of your data. We’ve made it even clearer what we do with personal data.
    • Data monitoring. You can audit how we’re doing with your data and request it back from us.
    • Data sharing. We’ve clarified whom we share your data with and why.
    • Security reporting. If there’s a breach of your data, we’re going to tell you and the appropriate regulators about it right away.

    For a thorough look at our new policies, check out our updated Terms of Service and Privacy Policy. As ever, you can write to us at privacy@camayak.com to object to any particular way in which we handle your data and we’ll address your concerns in the spirit these new protections are set up to encourage.

  2. Camayak scheduled downtime tomorrow and May 2nd

    ,

    Tomorrow and on May 2nd we’re going to be performing system maintenance, which we expect to take under one hour. You won’t be able to access Camayak during these periods so please make sure you’ve saved your work before we get started.

    Camayak may be accessible more quickly than we anticipate, so keep an eye on our Twitter account for updates on when you can resume your work.

    Depending on your location, here’s when Camayak will be unavailable:

    April 30th, 2018:

    • Central EU: May 1st 10 a.m. – 11 a.m
    • UK: May 1st 6 a.m. – 7 a.m
    • Eastern US: May 1st 1 a.m. – 2 a.m
    • Central US: May 1st 12 a.m. – 1 a.m
    • Mountain US: 11 p.m. – 12 a.m
    • Pacific US: 10 p.m. – 11 p.m
    • Hawai’i: 6 p.m. – 7 p.m.

    May 2nd, 2018:

    • Central EU: May 3rd 2 a.m. – 3 a.m
    • UK: May 3rd 1 a.m. – 2 a.m
    • Eastern US: 8 p.m. – 9 p.m
    • Central US: 7 p.m. – 8 p.m
    • Mountain US: 6 p.m. – 7 p.m
    • Pacific US: 5 p.m. – 6 p.m
    • Hawai’i: 1 p.m. – 2 p.m.

    We’re sorry for any inconvenience this causes and look forward to updating our Twitter account when we’re finished with our work.

  3. Spectre & Meltdown CPU patches

    ,

    Over the coming week Camayak will need to schedule short (up to one/two hours, but usually taking approximately 30 minutes) downtime windows with approximately 24 hours’ notice. This is in accordance with our hosting provider’s Meltdown CPU exploit vulnerability patches.

    We apologize for the short notice but due to the nature of the exploit downtime must be scheduled as soon as possible. Please see this blog post for a useful non-technical explanation of the two bugs and why the patches are necessary. Camayak may be accessible more quickly than we anticipate, so keep an eye on our Twitter account for updates on when you can resume your work.

    January 18th 2018:

    • Central EU: 3am – 5am
    • UK: 2am – 4am
    • Eastern US: 9pm Jan 17 – 11pm Jan 17
    • Central US: 8pm Jan 17 – 10pm Jan 17
    • Mountain US: 7pm Jan 17 – 9pm Jan 17
    • Pacific US: 6pm Jan 17 – 8pm Jan 17
    • Hawai’i: 4pm Jan 17 – 6pm Jan 17
    • Central EU: 8am – 9am
    • UK: 7am – 8am
    • Eastern US: 2am – 3am
    • Central US: 1am – 2am
    • Mountain US: 12am – 1am
    • Pacific US: 11pm Jan 17 – 12am
    • Hawai’i: 9pm Jan 17 – 10pm Jan 17
  4. Root Cause Analysis for Camayak Outage of February 3rd/4th

    ,

    Overview

    Camayak customers experienced three service outages though the evening hours of February 3rd (US Time Zones)/Early Morning February 4th UTC.

    The outages spanned the following times:

    Outage 1 (34 minutes):

    – UTC From 02/04/2015 03:39 to 02/04/2015 04:13

    – CST From 02/03/2015 21:39 to 02/03/2015 22:13

    Outage 2 (12 minutes):

    – UTC From 02/04/2015 04:37 to 02/04/2015 04:49

    – CST From 02/03/2015 22:37 to 02/03/2015 22:49

    Outage 3 (40 minutes):

    – UTC From 02/04/2015 04:54 to 02/04/2015 05:34

    – CST From 02/03/2015 22:54 to 02/03/2015 23:34

    In the times between the outages, service may have been slow as customers tried to reconnect.

    Part 1: The service outage

    A: Cause

    We have determined that the service outages were caused by the following circumstances:

    1) A customer inadvertently created a large number of publishing destinations pointing to non-existent or misconfigured WordPress instances.

    2) When the customer went to the manage publishing destinations page, Camayak attempted to validate the publishing destinations. Due to the specific circumstances of those WordPress instances, the validation code ended up waiting a long time for validation before timing out.

    3) The validation attempts tied up application server threads, preventing them from being used to service other customer’s requests

    4) Our load balancer, seeing that the applications servers were not responding to the validation requests in a timely fashion re-attempted those requests to other application servers, causing application server threads on those servers to be tied up waiting for validation.

    5) This process continued until all application server threads were tied up waiting for validation.

    6) Eventually, the validation requests timed out fully, and the application server threads were available to process other requests.

    Steps 2-6 of this process happened three times, causing the three outages.

    B: Remediation

    1) We have, as of February 4th 16:45 UTC (10:45 CST) released a fix to our application server software to ensure that reasonable timeouts on the validation attempts. The validation timeout is well below the load balancer timeout, to ensure the cascade effect in (4) does not happen.

    2) We have, as an additional measure, increased our application server capacity.

    Part 2: The support outage

    A: Cause

    The Camayak support team was unable to respond to support queries from customers, which began at UTC 04:01, until UTC 07:29. The reasons for this are detailed here.

    B: Remediation

    We have, as of February 4th 14:45 UTC (08:45 CST) set up a notifications system with extra features to alert and inform off duty support personnel of service outages. This is an addition to our existing notifications framework for other critical and non-critical alerts.

  5. How We Messed Up

    ,

    Last night we let our users down.

    At around 10pm Central Time, the Editor-in-Chief of the Kansas State Collegian emailed our support address to tell us that his staff had been unable to log into Camayak for ten minutes. Usually he would have expected a reply from us within the next fifteen, telling him what was up and what we were doing to fix it.

    Minutes later, the student media adviser at Pepperdine University and Editor in Chief of UCLA’s Daily Bruin both asked for help, too: Camayak wasn’t responding normally and had now been periodically inaccessible for at least twenty minutes.

    Screen Shot 2015-02-04 at 2.06.15 PM.png

    As anyone who uses Camayak knows, the whole point is to have real-time access to your newsroom, 24/7, no matter where you are. We work to maintain that standard because it’s an essential part of our service and also because people rely on us to produce their work on deadline.

    But last night, after thirty minutes of intermittent service, we still hadn’t responded to the people letting us know that we had a job to do.

    Screen Shot 2015-02-04 at 2.07.40 PM.png

    Then, minutes-that-felt-like-hours later and still without any support response from us, Camayak seemed to return to normal and newsrooms from around the country reported that their staff were able to keep working.

    Screen Shot 2015-02-04 at 2.09.56 PM.png

    But it didn’t last. Less than half an hour later, we received more alerts from other newsrooms of service interruptions that in some cases hadn’t let up since they’d first been reported.

    Screen Shot 2015-02-04 at 2.10.52 PM.png

    By this point it was abundantly clear that we’d let our users down on two counts. They’d been forced to deal with unplanned downtime and even more confusingly, were doing so without any guidance from us.

    Screen Shot 2015-02-04 at 2.11.57 PM.png

    Finally, at 11.45pm CST – a full two hours after the Kansas State Collegian had first reported difficulties accessing Camayak – we recovered full service.

    This morning, over an hour and half later, we sent our first responses to the people who’d reached out to us for help.

    Why we messed up

    First off, we’re profoundly sorry for letting our users down on two fronts last night. People use Camayak because it’s a more efficient way to work and we’ve always prided ourselves on our response times to any urgent difficulties our users are having.

    So what went wrong?

    Downtime

    Last night we experienced an outage, which we’re currently investigating. We’ll be updating our blog with our root cause analysis, later.

    Usually we’re in a position to react to such outages by monitoring our systems and adapt on-the-fly. Unusually, we didn’t respond like that last night. This is because we had a gap in our engineering and support coverage.

    Lack of support

    We’re used to ‘after hours’ questions from newsrooms, because that’s often when newsrooms get busy. We also work with newsrooms that are based in Pacific and other ‘later’ timezones. So the timing of last night’s issues was not outside of our predictable scope of activity.

    We’re a distributed company – which comes with timezone/support benefits – and decided three years ago that having a co-founder on the same timezone as many of our users was a sensible idea (we also have coverage for Central, Eastern and GMT/BST timezones). I’ve been living in Los Angeles ever since and am usually the first to respond to support questions from the Pacific timezones. We aim to get back to urgent requests within ten minutes.

    Last night however, I was not on the West Coast and our Central hub was off duty as our downtime began. As you’ll see from the time-stamps in this post, I’m currently in London, along with our GMT/BST hub that only came on duty at 6.30am local time (12.30am CST and 10.30pm PST).

    While these were highly unusual circumstances for our service and team, they shouldn’t have affected our service levels and we deeply regret that they did.

    What we’re going to do about it

    We’re going to publish the results of our root cause analysis for the outage Camayak experienced. Update: our analysis and remediations are listed here.

    Despite being spread across multiple timezones, we’re also taking steps to make sure our engineers and support team can be alerted to critical needs, even when they’re not on-call. The situation we ran into last night has never happened before but is a timely reminder for us to make sure we have a plan in place for urgent events that might occur when we’re all asleep or off duty.

    We’re hugely grateful for how responsive and patient our users are and won’t forget this as an example of how we can keep improving. We messed up and are determined not to let it happen again.

    If you’d like to follow-up with me, please reach out to roman@camayak.com.

  6. We Messed Up

    ,

    This evening, we messed up. Over the last four hours you may have been affected by several periods of Camayak downtime that have prevented you from logging in and doing all the things you rely on us to help you do. Camayak has been back up for the last hour but we have more questions to answer, which we’re going to do in a separate blog post, shortly.

    Update: you can read our follow-up post here.

  7. Introducing: In-line Comments

    ,

    In-line comments example.jpg

    One of the features that we’ve been really looking forward to adding to Camayak is a better way to give feedback on specificparts of assignments. After working closely with a number of editors and contributors, we’re excited to announce the release of our very own in-line comments.

    We made two key decisions in our implementation of in-line comments: that they should remain in the body of the assignment and that as a result, they should be collapse-able, to avoid them cluttering the reading/editing process. All in-line comments are saved as part of our revisions-tracking, so users can review in-line comments from previous versions. Best of all: when publishing directly to a CMS we automatically strip-out all in-line comments to prevent them from appearing in published copy.

    In-line comments, together with our recent API integration work with the Duke Chronicle, have also offered us the chance to move closer to our goal of allowing publishers to export their content from Camayak in whichever ways they require.

    Special thanks go to (among others) Danielle Muoio – Editor-in-Chief at the Duke Chronicle, Ashley Soley-Cerro – former EIC at Cal State Northridge’s Daily Sundial and Jillian Beck, EIC at UCLA’s Daily Bruin, all of whom were influential in guiding our implementation decisions.

  8. Introducing: Real-time Dashboard and Assignment Activity Feeds

    ,

    New dashboard.jpg

    This week we’re excited to announce two new features that will dramatically improve how teams communicate within Camayak.

    Dashboard activity feed. Each staff member will now be able to see a constantly updated feed of all the important activity happening in the desks they belong to. This will include: comments on assignments, new pitches, photos that are being uploaded to assignments, first-draft submissions, approved assigmnents and more.

    Assignment activity feed. You’ll now be able to review the entire history of all the important activity that’s occurred on and around an assignment, including: editor participation, photo uploads, new connected assignments and more.

    We like to evolve our features by listening to what our newsroom partners and over the last few months we’ve specifically asked our customers for their thoughts on how we could improve our navigation and alerts. Here are some of the comments we received, which we’ve sought to address with the new activity feed features:

    Photo Editors
    “We’d like to know who has uploaded a photo, without them having to add their name to the caption”

    “We’d like alerts for when someone uploads a photo to their assignment, so we can proof it quickly”

    Senior Editors
    “We’d like to easily see when someone has submitted their first-draft”

    “For approved assignments that we need to copy/and paste into InDesign, it would be awesome to see the assignment slug and author name displayed more clearly”

    “Can you show us new pitches as they come in, so we don’t have to open the pitches tab each time, to check for new ideas?”

    “It would be great to know what’s happening across all the assignments in my desks, without having to view each assignment one-by-one”

    Admins & Advisers
    “We’d love a full track-record of everything that’s happened to an assignment”

    “If I could log in and see everything that’s recently happened in our newsroom at-a-glance, it would help me know who’s been the most active recently and who to chase-up”

    Note: you’ll notice that the activity feeds introduce a few changes to the way you may be used to interacting with Camayak.

    – Activity items on each assignment will be displayed in the same list as comments and will now be ordered newest –> oldest (comments used to be ordered oldest –> newest).

    – Recently published assignments that used to appear at the bottom of the dashboard will be replaced by all approved assignments, appearing in a list next to the new dashboard activity feed.

    As ever, if you have any questions or feedback on the new activity feed features, please drop us a line at support@camayak.com. Enjoy!

  9. ,

    Wednesday, January 23rd 10:20 AM Pacific time, 18:20 UTC

    As of about 10:20 AM Pacific time, the network issues were resolved at our hosting provider.

    Testing and monitoring shows that Camayak is fully available again.

    We apologize for any difficulties the outage may have caused.  If there are any specific issues as a result of the outage, or otherwise, please contact us at support@camayak.com with details.

    Thank you,
    Camayak Support

     

    Wednesday, January 23rd 10 AM Pacific time, 18:00 UTC

    Camayak is experiencing significant packet loss and network slowness at our hosting provider.

    We are in contact with our hosting provider and are working towards a solution

    We will update this entry with more information as it becomes available.

    We apologize for these issues and are working to resolve them as soon as possible.

    Thank you.