Henk Nap

My day as a solver

At DBA.nl we are allowed to work from home, but I find working in the office more pleasant because of the collaboration with my colleagues. I log into our ticket management system and see that I have a number of tickets in my name today. I quickly run through it to check if there are any production disruptive issues. This is not the case and I will start with the first ticket in my name.

Ticket 1: Server crash
The first ticket is about a server that crashes about once every 2 months. The customer does not know the exact times, but after examining the xml files of the listener log, it appears to be at 22:03 during the week and at 6:04 at the weekend. Coincidentally, these are the default start times of the Oracle Automated Maintenance Jobs. After consultation with the customer, we decide to spread these jobs over the night. Should a crash occur again, the customer will contact us again.

Snack: User does not have sufficient rights.

Meanwhile, a call comes in at the service desk. A created user in the database does not have sufficient rights. The supplier is now with the customer and cannot continue. I can do that in between and I give the requested user extra rights. Problem solved and the supplier can continue.

Ticket 2: Performance problem

In this ticket, the customer indicates that a certain promotion has recently taken a long time. That is not enough information for me, so call the customer. It turns out that there is a significant growth in data in a short time and now the problem is that an action that was previously completed within 1 second now takes 10 seconds.

After provisioning statspack the client asked to perform the specific action again and in the statspack report I find a query that has a lead time of 9 seconds. This query does a full table scan on a table that has grown considerably. On a test environment I create an index and after that the query functions properly again. This ticket can also be closed. The customer will feed this back to the application supplier, who will take this further.

Snack: Evasion test
Then I get a chat message from a colleague. He is performing a fallback test at a customer based on a work instruction, but he gets an unexpected error message. Opening the standby database does not work for unclear reasons. It turns out to be a small database and there is also a backup available on the shared storage. Because there are people waiting to test, we decide to restore the fallback database from the backup. Fortunately, that works, so my colleague can continue.

Ticket 3: SR Oracle RAC

A Service Request is pending with Oracle due to a problem with the installation of the second node in a RAC cluster. Oracle requests to perform an action and to add its log files to the SR. I run the requested and upload the log files to MOS.

Ticket 4: OEM13c reporting

A colleague has previously returned from work at a customer and he has taken over a number of calls from me. That gives me time to continue working on an internal call to create a backup report that uses the RMAN Catalog in BI Publisher. The old Information Publisher report we used in OEM12c no longer works in OEM13c.

It’s already half past four, time to shut things down. Change your clothes and then get on your bike. I try to get to the office 1 or 2 times a week on my racing bike, which is a ride of about 1.5 hours.

Ticket 5: Patching Linux servers

In the evening at 20:00 a number of Linux servers still need to be patched. After applying the patches, the servers are restarted. Some databases are not showing up. These databases have been stopped and started once with sqlplus, Oracle Restart cannot handle this and does not start them during server start. I start the databases manually using srvctl, after that everything is up & running again. I pass this on to the client’s application manager. He starts the associated applications and does a short test. Everything works on his side too, the working day is now really over.