Keith Stevenson

Root Admin
  • Content count

    2,225
  • Joined

  • Last visited

  • Days Won

    5

Keith Stevenson last won the day on August 9 2016

Keith Stevenson had the most liked content!

Community Reputation

24 Excellent

About Keith Stevenson

  • Rank
    Senior Member

Profile Information

  • Gender
    Male
  • Location
    London
  • Interests
    SupportWorks, AssetWorks, Sci-Fi

Contact Methods

  • Website URL
    http://www.hornbill.com
  • ICQ
    0

Recent Profile Visitors

688 profile views
  1. All, This has now been resolved. Please check and confirm. We will provide a full explanation shortly Kind Regards Keith Stevenson
  2. Micheal, Thanks for the reply. Your instance is configured for us to Download email from your internal mail server via IMAP, so we would be connecting\pulling and uploading nothing (I suppose your router may count the commands we use to connect\view IMAP mailbox and download anything it finds as uploads but thats going to be far less than 37.4GB - probably < 10mb for entire day). Can you provide the report as a private message? Perhaps its the wording thats wrong? Kind Regards Keith Stevenson
  3. Micheal, Thanks for the post. Can you confirm where those stats come from? Kind Regards
  4. Prathmesh, Thanks for the confirmation. We noticed at 8:39 that one of our nodes was acting strange and refusing connections. Investigations showed that this was the result of a deadlock on the back end database. The lock was cleared around 8:48 and once done your instance could again connect. We are investigating the root cause and will add additional monitoring to ensure that this will not occur again. Kind Regards Keith Stevenson
  5. All, Following the above we have now made changes to the Analytics engine and Admin tool used to manage this (Prevents multiple joins that would return an extraordinary number of records). This is currently on our beta testing stream and once tested will be rolled out to all. We are also changing the way services handle large data sets (In excess of 30GB) and adding additional self correcting mechanisms to avoid issues (These will be rolled out as part of our continuous delivery of the weeks ahead) . Looking even further ahead we already have plans to overhaul reporting that will drastically reduce the impact of any large data sets. Kind Regards Keith Stevenson
  6. All, We have just witnessed the issue again (Same subset of customers). This occurrence of the issue has now been resolved. The root cause is that a set of temporary resources was exhausted causing a bottleneck whilst other requests waited. The resolve was to clear the block. This issue has become apparent due to a recent addition to the Service Manager application and advance analytics, which allows the creation of free form reports\joins. If these are not carefully crafted they can create unexpected large volume sets. We are working to identify any instance that may suffer from this issue and will be contacting the primary\secondary contacts for each instance to provide guidance on report creation whilst we investigate a more permanent solution. We apologies for any inconvenience. Kind Regards Keith Stevenson
  7. All, Thanks for the posts. We can inform you that our monitoring system detected an issue at 13:05 which effected a subset of instances . The root cause was identified and steps taken to resolve at 13:08, however it was then also found that this required a restart of 1 of the backened servers which took approximately 5 mins to restart. All services were fully restored at 13:18. We are still investigating the root cause and will provide a further update with analysis and steps we will undertake to ensure issue cannot happen again shortly. Kind regards
  8. Alex, Glad to hear that this is resolved. We have now identified the root cause (issue with stale Database connections getting reused rather than creating new ones) and will change our code to ensure that this doesnt happen again. We will also add additional monitoring checks to ensure that if such an event occurred in the future we are notified (and can take action) before it becomes an issue for yourselves. Kind regards Keith Stevenson
  9. Alex, We have cleared the blocked connections. Please try again and confirm that this is now OK. We are still investigating the root cause and will provide an update once we fully understand the issue. Kind Regards Keith Stevenson
  10. Gingib. Thanks for the post. No changes have been made to the SPF record since 9/15/2016 at 10:43:33 AM and having checked the SPF record everything looks as it should. Can you provide the full headers of an email that was sent and blocked so we can see where it got to. Kind Regards Keith Stevenson
  11. Reena, One option that might help whilst we review the possibility of changes would be to make use of a font designed for Dyslexia. This font (available from https://www.dyslexiefont.com/en/dyslexia-font/) is free for home\personal use and in Firefox (or chrome via extension) its a simple task to change all web pages to use this rather than ours. (See https://support.mozilla.org/en-US/kb/change-fonts-and-colors-websites-use) You can also change the background colour to help (Usually to Yellow) which should achieve your requirements. If the above helps im sure we can look at adding more options for accessibility in the future. Kind Regards Keith Stevenson
  12. Everton1878 Sorry to hear that your having trouble. There should be no issue as the DNS record for the above was fully propagated last week and the TTL on any old record should have passed easily by now. Can you do a NSLOOKLUP for service.hornbill.com from the CMD prompt on an effected machine and post the results as this will show us what DNS server your using, the records TTL etc. If you can also confirm this happens in all browsers that would help (Chrome does its own caching of DNS) Kind regards
  13. All, Following on from the above posts and as clarification. We monitor all service endpoints but have been monitoring them via their IP addresses and not using their FQDN (domain name), we also monitor our DNS service provider and to now that has worked ok. We made a decision to transition over to CloudFlare to take advantage of their superior service as well as future global caching to help accelerate our application content, this was executed last night. Unfortunately we misconfigured the DNS entry for service.hornbill.com but our monitoring continued to show the service as available because we were monitoring the service via its IP address and not its FQDN. We have now reviewed our monitoring setup and re-configured to use the domain name rather than IP address for all services to ensure this gap in our monitoring does not happen in the future. Sorry for any inconvenience caused. Kind Regards Keith Stevenson
  14. Adrian, Sorry to hear that your having problems. Ive looked at the server and although your instance was taking a large amount of RAM about 30 minutes ago, it seems to have settled down. From the logs I can see users logged it and performing actions. Can you confirm that you are still having issues? As for the reason why the RAM was high, the logs state that an analyst was processing an attachment with large number of attachments, but I will need our development team to investigate further. Look forward to your response. Kind Regards Keith Stevenson
  15. Ben, If you goto the Admin portal, under Home->System->Email-Templates you can click to edit the required template and then change the variable to mixed case ( ie H_Summary for mixed, h_summary for lower or H_SUMMARY for upper). Kind Regards Keith Stevenson