The (slightly tongue in cheek) role of the database administrator

-

As a former DBA, I find a disturbing trend toward a value proposition that is almost nonexistent among a recent crop of database administrators. Maybe having some background and/or working with other stellar DBAs in the past has spoiled me, but here's the workflow I've find more and more common.

scenario - production application has slowed down for a few transaction types, dynatrace shows a critical sql statement has slowed down. None of the development team has access to run explains, we can't "afford" hardware to load the production dataset into another environment (because we're using a DBMS that costs 13.6 bajillion dollars per CPU nanosecond with an additional upcharge of 1 million pounds sterling every time we execute a query that uses DML... Explains indicate everything is optimal in the lower environments. The decision to use this particular platform after the salesman for the product took the DBA team to Vegas for a "conference" and after tough negotiations (forever documented by the excellent 'based on a real life story' movie: "The Hangover").

Step 1 (optional) DBA team notices the query is slow also...email from DBA team to application team "Dear application team, while eating donuts and sipping single malt whiskey this morning, I accidentally hit a button on this funny looking device sitting on my desk and the monitor in front of me popped up a report indicating this sql is pretty slow, thought I'd let you know. Please fix ASAP as I believe we're wearing our our disk spindles and they're VERY expensive. Also, do you know how to close the window with this report...it overlaid the Rugby World Cup streaming live in HD and I really want to see the 'All Blacks' win!"

Step 2 (optional step one) email from application team to DBA team "Dear DBA team, we've (also) noticed this query is very slow. In every other environment it runs fine (less than 5 milliseconds), and it uses the primary key on every table, we're unclear why it takes 10 minutes to complete production. Can you investigate?"

Step 3 email from DBA team to application team "Dear Application Team - as I mentioned before, this query is slow and my children's college fund is contingent on our database using less than 20 IOPS under peak load, fix this IMMEDIATELY! upon investigation I think if you removed that query the system run much better...I will note that your application is causing our database to use a lot more cpu and IO than when we initially powered the system up. Obviously you don't know how to write software as prior to your application coming online the 3 'hello world' applications we used to test the database platform didn't cause any problems like this. Just because you collect data from sensors around the globe and provide real time data to thousands of users simultaneously doesn't mean you can just slap crappy SQL in and expect it to run well. Please let us know if you need any assistance writing application software as we're clearly much more intelligent than you."

Step 4 followup email from DBA team to application team "Dear Application Team - we further noticed you're heavily using the database during our backup window from 1am to 8am eastern. It's critical the system not be used during this time window as we have a team of interns backing up the entire database on 3.25" floppies. Please tell your users not to use the system during this window or remove these SQL statements ASAP. Also, do you know a good torrent client? I really need to catch up on the last season of 'Game of Thrones' PS did you see the latest news on CNN, evidently a lot of users of your system are complaining about about performance during peak online shopping hours in Europe...Hope you figure out what you did wrong when writing your application software."

Step 5 application team changes system to use BDB running on an old android device connected to the internet via a 2g wifi hotspot....sets up elaborate VPN solution to enable application servers to use this database from production data center...problem goes away

Step 6 entire DBA team is promoted for "saving the company millions of dollars by optimizing key SQL queries

Evidently, this is the new normal. Apologies to any DBAs who might, in fact, have been more helpful or proactive in helping to solve the problem.

Comments

Post a Comment

Popular posts from this blog

Please use ANSI-92 SQL Join Syntax

-
Image
Way back in 1992, a standard was published for SQL. In it, the syntax for specifying joins in SQL Database was finally standardized. This was a good thing because (for folks who've been doing this a while), trying to decode the vagaries of how various Databases handled outer joins could drive you bonkers yielding different results depending on the parse order of things in the from and where clauses. Unfortunately, some DBMS vendors took a while to adopt this standard (Oracle didn't support it until 2001), but at this point if you're using a major RDBMS and NOT using it, you're probably mired in a tradition or habit that bears changing. As an example of what I'm describing, here are some examples of what I mean. Old School (no standard syntax + implementation and evaluation of predicates varied by vendor [and query]): select * from customer, order where customer.customer = order.customerid ANSI way (agreed to standard, predicate evaluation has some rules ...
Read more

The difference between Scalability, Performance, Efficiency, and Concurrency explained

-
I thought I'd give some explanation of the difference between scalability, performance, Efficiency, and concurrency. To make things more accessible (and because I'm watching a cooking show on TV) I'm going to use a baking metaphor to help explain the differences. In our case, the system in question is baking cookies for consumers. Scalability, is how well something responds to adding more resources. If we can double our capacity by doubling the number of resources, we've got linear scalability. In our baking example, if we add more ingredients, we can scale to a larger number of cookies. To contrast with our other metrics, this doesn't actually increase the performance of creating a single cookie (they still take 10 minutes to bake), and the efficiency is the same (we just have more capacity). We also cannot bake more cookies in a fixed time period, but over a larger time period we can produce more cookies. Performance is how fast something can get done, u...
Read more

the myth of asynchronous JDBC

-
Image
I keep seeing people (especially in the scala/typesafe world) posting about async jdbc libraries. STOP IT! Under the current APIs, async JDBC belongs in a realm with Unicorns, Tiger Squirrels, and 8' spiders. While you might be able to move the blocking operations and queue requests and keep your "main" worker threads from blocking, jdbc is synchronous. At some point, somewhere, there's going to be a thread blocking waiting for a response. It's frustrating to see so many folks hyping this and muddying the waters. Unless you write your own client for a dbms and have a dbms that can multiplex calls over a single connection (or using some other strategy to enable this capability) db access is going to block. It's not impossible to make the calls completely async, but nobody's built it yet. Yes, I know ajdbc is taking a stab at this capability, but even IT uses a thread pool for the blocking calls (be default). Someday we'll have async databa...
Read more