On Call Troubleshooting¶

The on-call system functions basically as a first-tier of support: clients see more progress sooner, and tickets can be handled with a clearer understanding after being processed once by a worker. This is a list of troubleshooting tips to clarify that first pass. This does not supersede the on-call protocol (i.e. if unclear, ask if an issue is urgent). When a ticket comes in, claim it for yourself and acknowledge it (e.g. "Thanks for getting in touch with us! I'm going to do some initial research right now to assist in troubleshooting, and will report my findings when I escalate the ticket."). This process should involve about 15 minutes of research, after which the ticket should be passed along with your findings.

• What does successful resolution of this ticket look like?
  • Is this a bug/misconfiguration? Or a training issue?
  • Often a client self-reporting is reliable - but sometimes someone thinks there's a bug because they're misunderstanding the situation. Or they see unexpected behavior, but misunderstand what causes the problem.
• How can I reliably replicate the issue?
  • Attempt to create a test which triggers the problem every time.
  • When possible, don't just find a test that triggers the problem, but also isolate the problem as much as possible. If someone reports that certain web pages aren't loading images - which pages do and which don't? Test a bunch, look for the pattern that hints at the source of the bug.
• Restate the problem in more exact terms.
  • This is helpful both for internal documentation, but also if you need to pass the ticket off. For instance, a tech support client might call in and say, "My computer won't start up." That could mean:
    • "My computer won't turn on at all."
    • "My computer turns on, but the screen stays blank."
    • "My computer starts booting, but freezes at the BIOS screen (or "Starting Windows" screen, etc.).
    • "The computer gives a 'Blue Screen of Death' error during startup".
      ...or several other possibilities. Identifying which one is happening and restating it in technical terms helps everyone who touches the ticket.
• Find error messages. Know how (and when) to use:
  • the browser's Developer Console.
  • Know where to find logs for Drupal, CiviCRM, Apache at the very least.
  • Know how to enable debug modes. Civi debug/backtrace, Views show SQL, etc.
  • If you have error messages, make use of them.
  • In more advanced cases, you'll grep the codebase.