Three OnBase admins on running a business-critical system

01 06:54

100% said business critical. Then the job changes.

We opened with two polls. First, one word for what takes up most of your time as an OnBase admin. The two biggest words in the cloud were users and monitoring. Second, do you consider your OnBase to be business critical. Twenty-five responses, all yes.

Wayne’s first question was what changes when the system goes from important to business critical. Stephanie said it comes down to stability, insight into the health of the system, and acting before a user does. “I can no longer rely on a support ticket to come through to let me know that one of my critical processes is down.” If the application server is crashing, she wants to be the one putting up the splash page, not the one reading about it in a ticket.

Melissa said there are levels of mission critical. In healthcare, someone’s life is on the line if an EKG result can’t be viewed in the patient’s chart. In banking, people feel their money is mission critical, “but I think it’s less than dying.” At any level, your ability to make changes is severely reduced. “I never want to be in the news because of something I misconfigured.”

Kevin said a lot of admins aren’t ready for it. A departmental implementation grows until it’s running critical parts of the business, “and they didn’t know what they didn’t know.” That’s the step-back moment: do I have failover, or do I have to re-architect for what I’m responsible for now.

02 12:10

Governance around configuration is foundational to your ability to monitor

Kevin runs 700 active workflows. His team has naming conventions and monitors that they’re followed. When something breaks, and he said when, not if, you have to be able to tell which workflow and which queue you’re looking at. A workflow timer firing continuously through queues nobody can identify is a firefight you shouldn’t be having.

Melissa’s example was from that morning. A team was querying on a keyword that stores a source system’s identifier, sending six digits, and assuming they were unique. “You’re unique within your application. You are not unique across my whole enterprise.” Running an enterprise system means anticipating the problem six months or six years out.

Does it change in a regulated industry? Melissa: garbage in, garbage out, regulated or not. Regulation raises the fines. It doesn’t change the practice.

Stephanie’s team counted 700 import processes one day and 702 the next with no record of where the two came from. They rolled out process documentation standards and built their own change log.

Kevin’s summary is the clip I’d send to anyone who thinks monitoring is the starting point.

03 21:20

“Prior to about March, everything was reactive.”

Stephanie said that before March of this year it was all reactive. A user puts in a ticket, the team goes and figures it out. Hyland managed services watched some things, but not 24/7. With a large conversion coming that will multiply what’s in her system, that wasn’t going to hold. That’s when they bought Reveille.

What she monitors now: about 30 production servers, every polling location, and folders where documents sit longer than they should, which is the signal a service is hung. She’s building a decision tree so the restart happens automatically instead of alerting her. She also gets alerted when a sending system hasn’t delivered files, “because we’re still going to get the blame.”

04 25:22

The service is running. It’s not doing anything.

Wayne asked how they avoid alert fatigue.

Kevin has over 500 monitors, classified by severity. Informational ones, like a naming convention violation, notify once and produce a list that department admins use to clean up before migrating. Urgent ones are services down and workflow timers delayed. With 60 to 70 timer services, a delayed timer means the service is running but not doing anything, and documents aren’t coming in. “We don’t necessarily know that unless we know this should have kicked off at this time and it hasn’t.”

Melissa monitors import directories against her timer schedule. For documents aging in workflow queues she escalates in stages: the user for the first seven days, the whole user group for the next 14, and at 21 days the admin team and management. Those monitors open ServiceNow tickets automatically so first-level triage picks them up instead of the admins watching an inbox. Kevin’s point on all of it: “There’s a ticket created somewhere that has the information. That’s critical.”

A cloud OnBase customer asked in chat about recognizing hung processes in a hosted environment. Kevin said it’s hard to do what his team does if you’re hosted, but the data exists: expected and next runtime for timers, DIP, and COLD are all in the database, and the queries are standard. The hurdle is access.

05 37:20

The most common mistake, in one word: Document.

Wayne asked what the most common administration mistake is, the one that creates the most downstream work. Stephanie answered before he finished: “Document.” Kevin: “Documentation.”

Then Stephanie told the story from that morning. Users in one region kept requesting access to a group of documents. Access was granted, tickets closed, and they still didn’t have it. She took the ticket herself, added every user by hand, and logged in the next day to a long email from a regional manager. The users were gone from the group again. Someone had renamed the user group in OnBase so it no longer matched the IDP config, and every login removed them. Her new change log hasn’t surfaced who changed it yet.

An attendee asked what each panelist uses for change logging. Stephanie: a WorkView application one of her developers built, with full-text search over it. Kevin: reporting dashboards tied to monitoring, plus a database-level compare across all four environments, so he can see whether production matches test. Melissa: every production change goes through approval with step-by-step documentation, SQL validation of QA against production, and Studio compare for workflow. She wishes it compared properties and user groups better. “It’s really hard to sort through a 1,600-page document.”

06 46:48

“You’re not an island.”

Wayne’s last question was about the ecosystem around OnBase: database, infrastructure, Windows, integrations.

Kevin: where they can take control of the monitoring, they do. They’re an Oracle shop that depends on materialized views, so they alert if those aren’t refreshing. Fifty-seven servers, full access, API integrations monitored too. “When you’re enterprise, that means you’re integrated, you’re connected and depending on other systems. So it all has to be orchestrated.”

Stephanie doesn’t worry about the database; her DBAs find problems before she does. The Windows team is a different story, and she described reading them the riot act for ignoring alerts. Reveille now watches that server infrastructure for her. On one custom API server that hogs memory, it restarts IIS. On a custom mailbox importer that writes to the event log every second, it kills and restarts the service if the log goes quiet for five seconds. “I couldn’t tell you the last time we got a call because emails weren’t coming into OnBase. Before, it used to happen all the time.”

Melissa’s answer to the same question was four words: “Know what you’re changing.” Put it next to “Document” and you have the two habits every story in the session came back to.