Jump to content

Keith Stevenson

Root Admin
  • Content count

  • Joined

  • Last visited

  • Days Won


Keith Stevenson last won the day on August 17

Keith Stevenson had the most liked content!

Community Reputation

29 Excellent

About Keith Stevenson

  • Rank
    Senior Member

Profile Information

  • Gender
  • Location
  • Interests
    SupportWorks, AssetWorks, Sci-Fi

Contact Methods

  • Website URL
  • ICQ

Recent Profile Visitors

746 profile views
  1. Issues all over the place since last update

    Lyonel, Can you confirm that this is now working again. Kind Regards
  2. Unable to Access HB

    All At around 10:35 this morning, during a routine update to a backed server an error occurred in our subscription checker. This caused a number of instances to temporarily loose the count of available subscriptions for a subset of applications. The error was quickly identified and resolved by 10:37 (Change backed out as per change request plan). The type of update that occurred on the backend server happens around once a week as we continuously develop\enhance our service and this is the first time an error has been witnessed. The issue was not seen in our development or beta environments during testing. Inorder to prevent issues like this in future we are already looking at having additional tests in place. Kind Regards
  3. All, This has now been resolved. Please check and confirm. We will provide a full explanation shortly Kind Regards Keith Stevenson
  4. Extremely High Bandwidth from Mailbox to Hornbill

    Micheal, Thanks for the reply. Your instance is configured for us to Download email from your internal mail server via IMAP, so we would be connecting\pulling and uploading nothing (I suppose your router may count the commands we use to connect\view IMAP mailbox and download anything it finds as uploads but thats going to be far less than 37.4GB - probably < 10mb for entire day). Can you provide the report as a private message? Perhaps its the wording thats wrong? Kind Regards Keith Stevenson
  5. Extremely High Bandwidth from Mailbox to Hornbill

    Micheal, Thanks for the post. Can you confirm where those stats come from? Kind Regards
  6. unable to log on to Service Manager

    Prathmesh, Thanks for the confirmation. We noticed at 8:39 that one of our nodes was acting strange and refusing connections. Investigations showed that this was the result of a deadlock on the back end database. The lock was cleared around 8:48 and once done your instance could again connect. We are investigating the root cause and will add additional monitoring to ensure that this will not occur again. Kind Regards Keith Stevenson
  7. Hornbill Down?

    All, Following the above we have now made changes to the Analytics engine and Admin tool used to manage this (Prevents multiple joins that would return an extraordinary number of records). This is currently on our beta testing stream and once tested will be rolled out to all. We are also changing the way services handle large data sets (In excess of 30GB) and adding additional self correcting mechanisms to avoid issues (These will be rolled out as part of our continuous delivery of the weeks ahead) . Looking even further ahead we already have plans to overhaul reporting that will drastically reduce the impact of any large data sets. Kind Regards Keith Stevenson
  8. Hornbill Down?

    All, We have just witnessed the issue again (Same subset of customers). This occurrence of the issue has now been resolved. The root cause is that a set of temporary resources was exhausted causing a bottleneck whilst other requests waited. The resolve was to clear the block. This issue has become apparent due to a recent addition to the Service Manager application and advance analytics, which allows the creation of free form reports\joins. If these are not carefully crafted they can create unexpected large volume sets. We are working to identify any instance that may suffer from this issue and will be contacting the primary\secondary contacts for each instance to provide guidance on report creation whilst we investigate a more permanent solution. We apologies for any inconvenience. Kind Regards Keith Stevenson
  9. Hornbill Down?

    All, Thanks for the posts. We can inform you that our monitoring system detected an issue at 13:05 which effected a subset of instances . The root cause was identified and steps taken to resolve at 13:08, however it was then also found that this required a restart of 1 of the backened servers which took approximately 5 mins to restart. All services were fully restored at 13:18. We are still investigating the root cause and will provide a further update with analysis and steps we will undertake to ensure issue cannot happen again shortly. Kind regards
  10. Failed to initialize application - operation timed out

    Alex, Glad to hear that this is resolved. We have now identified the root cause (issue with stale Database connections getting reused rather than creating new ones) and will change our code to ensure that this doesnt happen again. We will also add additional monitoring checks to ensure that if such an event occurred in the future we are notified (and can take action) before it becomes an issue for yourselves. Kind regards Keith Stevenson
  11. Failed to initialize application - operation timed out

    Alex, We have cleared the blocked connections. Please try again and confirm that this is now OK. We are still investigating the root cause and will provide an update once we fully understand the issue. Kind Regards Keith Stevenson
  12. Outgoing emails being blocked by mail control

    Gingib. Thanks for the post. No changes have been made to the SPF record since 9/15/2016 at 10:43:33 AM and having checked the SPF record everything looks as it should. Can you provide the full headers of an email that was sent and blocked so we can see where it got to. Kind Regards Keith Stevenson
  13. Accessibility requirement - Change field colour on assets

    Reena, One option that might help whilst we review the possibility of changes would be to make use of a font designed for Dyslexia. This font (available from https://www.dyslexiefont.com/en/dyslexia-font/) is free for home\personal use and in Firefox (or chrome via extension) its a simple task to change all web pages to use this rather than ours. (See https://support.mozilla.org/en-US/kb/change-fonts-and-colors-websites-use) You can also change the background colour to help (Usually to Yellow) which should achieve your requirements. If the above helps im sure we can look at adding more options for accessibility in the future. Kind Regards Keith Stevenson
  14. service.hornbill.com DOWN

    Everton1878 Sorry to hear that your having trouble. There should be no issue as the DNS record for the above was fully propagated last week and the TTL on any old record should have passed easily by now. Can you do a NSLOOKLUP for service.hornbill.com from the CMD prompt on an effected machine and post the results as this will show us what DNS server your using, the records TTL etc. If you can also confirm this happens in all browsers that would help (Chrome does its own caching of DNS) Kind regards
  15. service.hornbill.com DOWN

    All, Following on from the above posts and as clarification. We monitor all service endpoints but have been monitoring them via their IP addresses and not using their FQDN (domain name), we also monitor our DNS service provider and to now that has worked ok. We made a decision to transition over to CloudFlare to take advantage of their superior service as well as future global caching to help accelerate our application content, this was executed last night. Unfortunately we misconfigured the DNS entry for service.hornbill.com but our monitoring continued to show the service as available because we were monitoring the service via its IP address and not its FQDN. We have now reviewed our monitoring setup and re-configured to use the domain name rather than IP address for all services to ensure this gap in our monitoring does not happen in the future. Sorry for any inconvenience caused. Kind Regards Keith Stevenson