RSS

Tag Archives: Lessons

Thinking outside the Box in Disaster Recovery


I had an interesting ‘aha’ moment awhile back.

I was trying to take a well-earned day off from work, visit my mom on her birthday, and in general, do a little decompressing.

I’d recently taken her portrait (a profession from a former life) and my goal was to give it to her and later take her to an appointment and then lunch.  We were between the appointment and lunch when my phone buzzed.

It was one of my Faithful Minions from the land of Eva Perón, who was in a bit of a bind.

Seems that a server had a database that had gone corrupt.

No, not the Chicago style “vote early, vote often” corruption.

Database level corruption.

It seems that this server had been giving us some trouble with backups (it didn’t want to back certain databases up, depending on what node of the cluster it was on) – and the team responsible for fixing that had been working on it for a few days – with limited success.

Aaaand another one of my Faithful Minions, from the land of Chicken Curry, had tried several things, one of which included detaching the database.

Which worked.

It’s just that once you detach it, you likely want to reattach it.

But trying to reattach a database that’s been munged to some degree is a bit of an issue, as SQL tries to recover the database and chew through the transaction log file prior to bringing it online.

That’s as it should be.

Problem is, this db had some level of corruption that would cause an issue when it got to 71% recovered. Stack dump, the whole nine yards.

It was, in a word (or several) a perfect example of a disaster recovery exercise. Yay! I’d always wanted to have one of my very own…

(well, not really, and not on my day off, and…)

Hmm…

I gave my Faithful Minion from the land of Perón some instructions while sitting in the car in front of the restaurant Mom and I were going to have lunch at, and then tried to enjoy birthday lunch with her, but still had my mind being pulled dripping out of the clam chowder and back to the database while Minion worked on his Google Fu a bit to try to figure the error out. Mom and I finished up lunch and I took her home, where I logged in to see what Minion and I could accomplish.

So the thing is, we use DPM (part of Microsoft’s System Center suite of products) to do backups. That’s Microsoft’s Data Protection Manager, a program – no – a system that really takes some getting used to. There will be another post in this series about it – but one of the big things you have to get used to as a DBA is that you don’t do backups anymore…

You don’t use Maintenance Plans.

You don’t use Ola Hallengren’s or any of the other amazing tools out there to handle backups.

You really don’t do backups any more.

DPM does.

No backup jobs on your servers…

No backup drives…

No…

Backups…

DPM handles them all semi-invisibly…

It is – well… weird.

It took me a long time to wrap my head around it.

But it works.

And that’s where this situation kind of got a little strange.  (Well, strange*r*)

See, DPM creates Recovery Points (its version of a backup) and it will stage the backup on disk either locally on the SQL box or in the DPM system where you have MDF and LDF files created with some sort of wizardry before it’s shipped off to tape.

So, we poked and prodded, and did all sorts of technical things, until my next set of Minions from The Pearl of the Orient came online – and we tried to find the backup of the database from DPM.  This took a good while.

Longer than we were expecting, actually.

While they were looking and I was troubleshooting, much time had passed in the land of rain (a bit south of Seattle) and lunch was long, long past.  Mom made some dinner while I was working on my laptop, sitting in my dad’s old chair, while I said goodbye to and thanked a very tired Minion in the land of Perón,   Meanwhile, Minions in the Pearl of the Orient, and Minions (and a Minion in Training) in the land of Curry were all trying to help, trying to troubleshoot, and in the case of Minion in Training, trying to learn by soaking up individual droplets of knowledge from a fire hose of a real live DR exercise.

Which is where it got interesting…

See, with DPM, you can choose a recovery point you want to restore to, and what it will do is simply replace the MDF and LDF files on your SQL box with the ones it’s reconstructed in all its magic.

The good news about that? Once you’ve learned how DPM works, anyone can pretty much restore anything that’s been backed up. That means Exchange servers, file servers, SQL servers, you name it…

It is (and I hesitate to use this word at all) kind of cool.

You need a restore?

All you have to do is open up DPM, find the database you need…

<click>

<typety type>

<find the backup that happened just prior to whatever blew up.>

<click> <click>

<click>

<pick the recovery point you want>

<click… scrollllllllll…. Scrollscrollscroll <click> <click>

Then you wait till DPM says it’s ready.

And tadaa…

That is…

If DPM had backed that database up.

But for that particular database…

On that particular node…

Of that particular cluster…

That database had not been backed up.

By DPM.

For… Let’s just say it was out of SLA, shall we?

It was truly a perfect storm.

The team *had* been working on it – it just hadn’t gotten actually fixed yet, and the crack that this database had slipped through grew wider and wider as it became clear to us what had happened.

It’s the kind of storm any one of us could run into, and this time it was me.

We looked high and low for a DPM backup set, and found one on that was supposed to be on tape.  But, given that this was Thursday where I was and Friday where various minions and co-minions were, the earliest we could get the tape would be by Monday…

More likely Tuesday…

On a scale of one to bad, we were well past bad.  We’d passed Go, had not collected our $200.00 – and…

Well, it was bad.

Several days’ worth of waiting on top of the existing time lost wasn’t really going to cut it.

Then, Co-Minion found that there was a DPM backup on disk – but a day older than the one on tape.

I could have that one in two hours (the time it took to copy the mdf/ldf files over and then put them in the right location.)

Hmmmm. A backup in the hand is worth two in the – um… Tape library?

So we copied it over.

And I attached it…

And now we had a database on the box that worked.

Yay, system was up and running.

But…

it wasn’t the database I wanted.

Like any of you out there, I wanted the backup from just before the database had blown up, and because of all the troubleshooting that had happened before I got involved, some of which actually made the situation quite a bit more challenging, it turned out I couldn’t do anything with the files I had… so reluctantly, I had to go with the older files, simply because I could get them quicker.

And the next day, while trying to fix another issue on that cluster, I got to talking to the guy who runs the SAN and the VM’s, and I explained what all we’d gone through over the course of getting that database back, and he said, “Wait, when did you need it for?”

I told him.

“Oh, I’ve got copies on the SAN of those drives – we could mount it if you need – 12 hours ago, 24 hours ago, whatever…”

He… You…

What?

I realized what he was telling me was that if I’d contacted him (or, frankly, known to contact him at that time of night) – I could have had a much more recent backup sooner instead of spending so many hours trying to fix the existing, corrupt database.

What it meant, was I was thinking inside the various boxes I was dealing with…

Trying to get a *SQL* backup…

Trying to get a *DPM* backup…

When what I needed, frankly, was bits on disk.

I needed MDF and LDF files.

Not only could DPM get me MDF and LDF files, but so could the SAN.

And the SAN backups had them.

I just didn’t know it.

By the time I found out – the old database we’d attached had had data written into it and had been up for a good while. Merging the two would have been far, far more of an issue than it was already (I’d had experience with that once before as an end user) .

So where does this leave us?

It means that if you’ve got a SAN as your storage mechanism that’s backing itself up, you might find yourself thinking outside the box you’re used to thinking of when trying to restore/recover a database.  Go buy the person or people running your SAN a cup of decent coffee and just chat with them, and see if this kind of stuff is possible in your shop.

They might be able to help you out (and save you substantial amounts of time) in ways you quite literally couldn’t have imagined.

In the end, I learned a lot…

My team learned a lot.

My mom learned a lot.

I will be making some changes based on what we learned, with the goal of being able to have some more resiliency in both the hardware resources we have as well as my faithful Minions who worked hard at trying to fix things.

Ultimately, one of the takeaways I got from this was something simple, but also something very profound.

Make sure the stuff you’re doing in your day job continues to work, so you can actually have a day off to spend with your loved ones when you need it.

Oh – and the portrait of mom? She loved it.

It’s here:

The portrait of Mom I wanted to deliver to her.

The portrait of Mom I delivered to her.

 

Take care out there, folks…

Tom

 

 
Leave a comment

Posted by on April 1, 2016 in Uncategorized

 

Tags: , , , , , ,

Adventures in Fixing VLF issues on a database in an Availability Group, backed up with DPM, with multiple secondaries


Hey all,

We had a server failover recently during a demonstration to one of the C-level folks, and as you might imagine, it was brought to my attention. The reason for the failover wasn’t something I could control, but the length of time the relevant database took to come up was something I could – so when it finally came up, I noticed the errorlog with this little note:

Database <dbname> has more than 1000 virtual log files which is excessive. Too many virtual log files can cause long startup and backup times. Consider shrinking the log and using a different growth increment to reduce the number of virtual log files.

Hmm…

More than 1000, huh?

Slow recovery from failover, huh?

Hmm…

Where had I seen this before?  Oh yeah, some years back, when we were doing datacenter maintenance, and the *one* sharepoint database  that had all our documentation on (ahem) “How to bring up a datacenter.”  in it took a half an hour to come back – I remember now!

So if you don’t know – having a metric buttload (note – that’s a real term there, go look it up) of virtual log files means that every time your server fires up a database, it has to replay the transaction log to get all up to date and everything, and if the transaction log’s got a bunch of virtual log files, it’s like trying to run data backwards through a shredder… it takes time, and that’s the issue.

I remembered Michelle Ufford’s ( blogtwitter ) post that I’d found back then about counting Virtual log files and tried it on this server now that it was up.

And got this nice little message:

Msg 213, Level 16, State 7, Line 1

Column name or number of supplied values does not match table definition.

Back to “Hmmm…”

Well, I knew dbcc loginfo(dbname) from her script was inserting into a table that the script created, and it had worked before. It had been awhile since I’d run it – so I figured maybe there was some change in what it returned based on the SQL version. So I decided to see what it was inserting…

Well hey – the original script was missing a field (first column above)

So I added it (RecoveryUnitID – the first one below)

…and it worked fine.

Turns out SQL 2008R2 (where the original script worked) returns different fields than 2012 and 2014 (where it didn’t).

I figured I didn’t want to find out which version of the script to use every time I needed to run it on a server, so I told the script to figure that out by itself, and then run the appropriate hunk of code (example below)

So what you see in that code is a combination of this:

https://msdn.microsoft.com/en-us/library/ms174396.aspx from Microsoft to check the version number,

and this

http://sqlfool.com/2010/06/check-vlf-counts/ from Michelle Ufford.

A couple of things to understand here:

  1. This code won’t fix the problem (read Kimberly’s posts below for that)
    1. http://www.sqlskills.com/blogs/kimberly/transaction-log-vlfs-too-many-or-too-few/
    2. http://www.sqlskills.com/blogs/kimberly/8-steps-to-better-transaction-log-throughput/
  2. It will, however, show you that a problem exists.
  3. …that you have to fix and manage (Back to Kimberly’s posts)

You’ll want to do some fixing that involves:

  1. Lots of log backups and log file shrinking, then log growing (See Kimberly’s posts for more info on this)
  2. Adjusting the auto-growth setting from the 1 mb/10% default setting to something significantly larger (Meanwhile, back in Kimberly’s posts… Seriously, check them out)
  3. Oh, and make those adjustments on the Model database so that every new database you create has decent growth patterns instead of the 1 mb/10% bit)
  4. Tadaa – you should find much faster failovers on your clustered systems (assuming you’re still using them and not Availability groups)

So… that made it look easy, right? Well, let me give you the Paul Harvey Version, “The Rest of the Story.”

What I learned, and what complicated things a bit…

My system was an Availability group with two nodes running synchronously, and two nodes in two different datacenters running asynchronously.

They’re backed up by Microsoft’s Data Protection Manager (DPM) that is configured to back up the secondary node.

So my plan was to back up the secondary – only when I did, it told me that the last recovery point (what a backup’s called in DPM) had something to do with Marty McFly, namely that it had been backed up in the future – about 8 hours into the future.

Hmm… Okay, that told me the secondary it was backing up was one of the asynchronous ones, and it was either in the land of fish and chips, or the land of beer and pretzels, either way – it was a weird way to think about trying to preserve the recovery chain.

I created recovery points through DPM, which did it on one of those secondaries, which, strangely, helped to clear the log on the primary.

Then I shrunk the log, in stages, on the primary, which then replicated out to the various secondaries locally and across the pond.

I was able to get the VLF count down from well north of 33,000 to just over 600. The log wouldn’t shrink any more than that, which told me there was a transaction hanging on toward the end of one of the the virtual log files, and frankly, for right then, that was okay.

To fix it further, I almost entertained the idea of flipping the recovery model from full to simple, clearing the log file completely, then flipping it back to full, but that would have totally broken my recovery chain, and with three secondaries, I had no desire to try to fix all the fun that would have created. (Translation: Rebuilding the availability group – Uh… No.)

I then grew the log file to an appropriate size for that database, then increased the growth rate on the database files to something far more in line with what Kimberly mentioned above.

All of this was done on the primary, and the changes I made propagated out nicely, so I only had to make them in one spot.

Oh, while I was researching this…

(I’d done this before – reducing VLF counts, but never on an Availability Group, much less one with multiple secondaries in different time zones where DPM is backing up one of the secondaries, where it’s not obvious which one of the secondaries is the one being backed up)

…I noticed other people had clearly run into this same issue (reducing VLF counts) – and had not only entertained the idea of automating VLF reduction, but taken the idea out drinking, dancing, and somehow ended up with a receipt for a cheap motel room – and then put it all into a stored procedure.

That ran…

In a job…

On a schedule…

Um.

I decided that breaking the recovery chain was bad enough, but shredding it was so much worse – so this part will not be automated.

At all.

Oh – and here’s the code. It’s Michelle’s, with a couple of tweaks from me and one from Microsoft.

--Code to determine how many virtual log files there are in a database.
--modified from the original code here: http://sqlfool.com/2010/06/check-vlf-counts/
--to check for multiple versions of SQL because of schema changes between versions
--if any of the results are > 1000, do multiple log backups (or DPM Recovery points
--to clear out the transaction log + shrinking the tlog to get rid of
--the extra virtual log 'fragments'.
--As always - test this in your test environments first.  Your mileage may vary. Some 
--settling of contents may have occurred during shipping.  You know the drill folks.
--Best reference out there for managing this:
-- http://www.sqlskills.com/blogs/kimberly/transaction-log-vlfs-too-many-or-too-few/
-- http://www.sqlskills.com/blogs/kimberly/8-steps-to-better-transaction-log-throughput/
--the code below handles SQL 2008R2, 2012, and 2014. If you're running something other than that,
--just run a dbcc loginfo(YourDatabaseName) and make sure the #stage table has the same
--columns in it
--if it doesn't, add a section below.
DECLARE          @sqlversion SYSNAME
SET              @sqlversion =(SELECT CONVERT(SYSNAME,SERVERPROPERTY('ProductVersion')))
PRINT            @sqlversion
-- SQL 2008R2

IF @sqlversion LIKE '10.50.%'
     BEGIN
          CREATE TABLE #stage
                                  (
                                     FileID        INT
                                   , FileSize      BIGINT
                                   , StartOffset   BIGINT
                                   , FSeqNo        BIGINT
                                   , [Status]      BIGINT
                                   , Parity        BIGINT
                                   , CreateLSN     NUMERIC(38)
                                  );

          CREATE TABLE #results
                                  (
                                     Database_Name SYSNAME
                                   , VLF_count     INT
                                  );
          EXEC sp_msforeachdb  N'Use [?];
                                Insert Into #stage
                                Exec sp_executeSQL N''DBCC LogInfo([?])'';
                                Insert Into #results
                                Select DB_Name(), Count(*)
                                From #stage;
                                Truncate Table #stage;'
         SELECT         *
         FROM           #results
         ORDER BY       VLF_count DESC
         DROP TABLE     #stage
         DROP TABLE     #results
     END
     --sql 2012/2014 - note - new temp table names because sql thinks old ones are there.
IF @sqlversion >= '11'
     BEGIN
          CREATE TABLE #stage2
                                  (
                                     RecoveryUnitID INT
                                   , FileID         INT
                                   , FileSize       BIGINT
                                   , StartOffset    BIGINT
                                   , FSeqNo         BIGINT
                                   , [Status]       BIGINT
                                   , Parity         BIGINT
                                   , CreateLSN      NUMERIC(38)
                                   );
          CREATE TABLE #results2
                                  (
                                   Database_Name SYSNAME
                                   , VLF_count INT
                                   );
          EXEC        sp_msforeachdb N'Use [?];
                                     Insert Into #stage2
                                     Exec sp_executeSQL N''DBCC LogInfo([?])'';
                                     Insert Into #results2
                                     Select DB_Name(), Count(*)
                                     From #stage2;
                                     Truncate Table #stage2;'
          SELECT         *
          FROM           #results2
          ORDER BY       VLF_count DESC
          DROP TABLE     #stage2
          DROP TABLE     #results2
     END

Thanks to the #sqlhelp folks for their advice, suggestions, good-natured harassment, and ribbing:

Allen McGuire (blog | twitter)
Argenis Fernandez (blog | twitter)
Tara Kizer @tarakizer (blog | twitter)
Andre Batista @klunkySQL (blog | twitter)

and of course, the ones whose reference material I was able to use to put this together:

Michelle Ufford (blog | twitter)
Kimberly Tripp (blog | twitter)
Paul Randal (blog | twitter)

<outttakes>
When I told Argenis about the DPM involvement and secondaries in multiple timezones, his response 🙂

 
4 Comments

Posted by on January 5, 2016 in Uncategorized

 

Tags: , , , , ,

Paul and the Butterfly Effect


So now that most of you reading this have come back home from either PASS Summit or MVP Summit and are trying to get your heads around being back home and/or in the office again, so grab a cup of coffee and allow me to share a story with you…

(for those of you who have no idea what I’m talking about – imagine a bunch of introverted computer geeks getting together, eating together, drinking various liquids together, learning together, and quite often having an amazingly good time while they’re at it)

This story started out as a response to a note Paul Randal (blog | twitter) wrote just after summit almost a year ago, and for any of you who’ve read my stories, you’ll be familiar with the phrase, “And it got me thinking” – usually about halfway down the story.  But this time it starts at the beginning, see, Paul wrote about this thing called “the butterfly effect” and he was talking about how making one small change in something could make major changes later on…

The links above are Paul’s, the links below will likely reference my own stories, which you’ll understand as you read along.

Paul asked for feedback on what he wrote about the butterflies, and I started writing, and thinking, and thinking, and writing.

And it went something like this:

Gosh, Paul,

It’s hard to figure out where the first butterfly flapped its wings in this story…

And please forgive me – it turned out longer than I expected, but you’ll understand when you get to the end.

I was working at Microsoft (1996-2000), and over time had sent out stories about my then 7 year old son (like this one) – and one day, I got a surprisingly snarky reply-all response back from a fellow I’d considered to be a friend.  We’d worked together (he in England, me in Redmond) – and while I wanted to fire back with all the self-righteousness in the world, how he, a young, single man, didn’t understand the joy that comes from having the privilege of having children, I backed off.

I checked with a trusted friend for advice, and then suggested to him, kindly, that in time, he would indeed understand the joy one can have experiencing a story like the one above, but he was not yet old enough to understand that.  I wished him well, and told him that I hoped that someday he would be able to experience that privilege.  I made sure not to burn any bridges, because his comment was based on youth and inexperience, and I truly valued his friendship.

Fast forward 10 years.

We’d both grown, we’d both lived, but most importantly, we’d stayed friends.

He’d moved to the US, worked for several companies, including the one I work for now, and is now back at Microsoft, based out of Chicago.

Some time back, he heard I was looking for work, and within a day, had written a letter of recommendation based on years of trust, years of both personal and professional friendship, and in part due to that – I had an opportunity to interview and was subsequently hired.

I had/got/made/took opportunities to speak (I did sessions down at SSWUG.org for a few years) and then got involved in the SQL community, starting, as I recall, with Chris Shaw (blog | twitter ).  He was one of the speakers at one of the SSWUG sessions at the time (I didn’t know about PASS then), and I was just being me, flipping him crap, and he flipped it back.  We laughed, then he asked if I wanted to speak.  I couldn’t imagine him asking that, but he did, and the conversation went like this:

“Have you ever done any public speaking?”

“Only Eulogies.”

<crickets>

“No really.  One was for my dad, and one was for a friend in my cancer survivor’s support group.”

I couldn’t imagine anything more stressful than doing a eulogy for your own dad, or for a friend who had died of the same disease you were fighting, too, so I figured I’d give it a shot.  How hard could it be, right?

Little did I know – but I worked hard, made some presentations, and I did it for three years. I met wonderful people there (Chris Shaw & Wendy Pastrick  ( blog | twitter ) were with me in the studio one of those times) and met others through the sessions.

Wendy Pastrick, me, and Chris Shaw

Wendy Pastrick, me, and Chris Shaw

…and Paul, I had a ball doing it.

This year I spoke at the SQLSaturday in Portland just before Summit – on communication, and how important it is, and how hard it is to do well. (I have a Bachelor’s degree in Communication and a Master’s in Visual Communication (photojournalism) and am still learning)

And it got me thinking, as so many stories do…

Had I gotten mad at John (the friend in Chicago) and burned that bridge way back in 1997 when I sent that original emailed story out (I think it says 1998 in the one on the blog) – then I wouldn’t have had him as a resource to get the job I have today.

Had I not gotten that job, I wouldn’t have had the speaking opportunity in Tucson.

…nor would I have gone to Summit.

Or had the opportunities to speak.

Or made the friends I’ve made.

I wouldn’t have realized there were other people just as lonely as I was out there who worked in their own little cubicle, being one of very few people in their companies doing what they do.

I wouldn’t have learned about #sqlhelp, and #sqlfamily, and summer camp for geeks (Summit)

I wouldn’t have learned that just by tweeting something with DBCC in it, in short order you could get an answer from the guy who wrote it.

As a result, my mind has been in a state of continuous bogglement (if that’s a word) for the last 7 years.

(you realize this list could go on for a good long while.)

But I did learn – and I know a little about these things now.

I know there are folks out there who will help, who will encourage, and who will cheer me on should I need it.

Just today, I found myself on the cheering/encouraging end of that equation (one member of the SQL community came home from Summit to one less family member in the house)

I don’t know if in doing that, I was a butterfly helping someone in their own life journey. I don’t know..

And I don’t think any of those things (that family member excluded) would have happened had I gotten mad at John those many years ago.

Yes, I do think about that.

Oh, speaking of John… He and his wife now have two boys – and he understands.

Take care Paul – thanks for making me think – that was fun.

Tom

PS: not to overload you, but to take the butterfly back even further – almost 100 years ago, there was this little piece of Russian Shrapnel

if it hadn’t hit where it did, you wouldn’t be reading any of this.

And I sent it…

And just now – it got me thinking some more…

Some of the butterflies in life are good ones.  Some are bad ones.

All of them got you to where you are today.

So I’ll end this one, uncharacteristically, with a question:  As you think back, What are your butterflies? What got you to where you are today?

Take care out there, folks,

Tom

 
Leave a comment

Posted by on November 6, 2015 in Uncategorized

 

Tags: , , , , , , ,

Life Lessons from SQLSaturdayABQ – Including one from Bugs Bunny


SQLSaturdayABQ

So it’s been a couple of weeks – but SQL Saturday ABQ has finally simmered long enough for me to write about it. I learned so much about so many things down there, and am tremendously grateful for the opportunity to share not only a meal and some learning with professional colleagues, but also reconnect with old friends and make new ones.

The trip down was great – I found out that putting my phone into Airplane Mode seemed to put some pretty cool Airplanes into pictures of an already gorgeous landscape.

I learned the Albuquerque is at 5,000 feet, and for someone used to living at sea level, I learned to appreciate the simple things in life, like, say, air.

(On our second day there the wonderful friend we were staying with took us up to the Sandia Peak Tramway. She and my wife enjoyed the gift shop at the lower elevations while I, still getting used to the 5,000 foot elevation,  went up another 5000 feet on the tram.

No, no oxygen masks fell out of the sky, but I could definitely feel all 10,378 feet of altitude there.

After I got back down, we explored some more, stopped at some lower elevations, and got a little perspective on the mountains for a little further north,

We stopped in a little town called Bernalillo and saw the Coronado Historic Site (above), and could hear the Rio Grande River gurgling down below. It was so different from what we have here in Washington – one could easily say it’s ugly and brown, but that would be missing the point – it’s got a beauty all its own, and needs to be looked at with different eyes.

As for the sessions with SQL Saturday itself…

I learned things about PowerShell in the session from Mike Fal ( b | t ) that made me want to get my hands dirty and try to find problems we’re struggling with in our environments that could be solved with a little PowerShell, and he proved that yes, you can indeed type in a demo, and then promptly demo’d why not to do it. J I loved the examples he gave, and the fact that he stuck with “learn the concepts, don’t freak about the code” – in large part because, just like in Field of Dreams (if you build it, they will come), with PowerShell, if you learn the concepts, the code will follow. You just have to understand what you want to do first – and that happens with the concepts. I’ve learned that if you give someone a problem first, and then give them a pile of tools, they’ll figure stuff out, and you’ll see creative juices flowing as they start thinking about new ways to solve old problems. It’s kind of fun to watch, and more fun to be part of.

I didn’t take any pictures of Jason Horner’s (b|t) session, but I enjoyed his presentation very much, which was full of demos and examples of databases much bigger than the ones I handle. As I recall, there were 4 (FOUR) MCM’s in the room… We were walking on hallowed ground there – and just that level of conversation, questions, and knowledge was fun to be around. I look forward to seeing more of his presentations, and applying what I learned there.

John Morehouse’s ( b | t ) session on social media had an interesting cross section of people in it – some quite experienced who are used to it, but always looking for new things to learn, and some who were absolutely new to the game and had never, ever used it.

John talked about how it can benefit you professionally, how easy it is to blur the lines between personal and professional, and how to do your best to keep them separate if needed. He made a very valid point that no matter where you are, you’re an ambassador for your company – so “think first, then post” along with the idea that once you hit send, it’s out there. Know your company’s policies on social media. That’s a huge thing. Even if you think it’s a private message, the wrong screenshots in the wrong places can be embarrassing for a long time. We also had a live demo of twitter, and how to do everything from getting a question answered with #sqlhelp, to getting a job via social media.

And I learned a lot – I got better doing my presentation on “Life Lessons in Communication” for the second time – and finally getting the butterflies in my stomach to at least fly in formation. I learned a lot from my audience, and had something totally off script pop into my head during the presentation – with the simple sentence of, “Are you solving the problem? Or managing it?” I realized I learned as much from my own presentation and audience as they did from me. Oh – and lessons are everywhere. You don’t even have to look hard. You just have to pay attention.

It was a lot of fun.

Many, many thanks to Meredith ( b | t ) and crew for getting everything together, and for the absolutely wonderful speaker’s dinner the night before. (what was that smokey salsa-y stuff on the chips? that was amazing!)

What came next was a time that can only be described as a slice of heaven.

Any of you in the IT industry know that the whole work/life balance thing is something that has to be managed very, very deliberately.  The cost of not managing it can be ridiculously high.

And so, for the next few days, I was able to spend that precious thing called time chatting with my wife and our friend, getting incredible amounts of fuzz therapy from two wonderful dogs, and just spending time away from the computer.  I allowed myself the time to absorb some of the lessons I’d learned at SQLSaturdayABQ, and then, as I watched some of the hot air balloons drift by, I realized that not all of the lessons I learned there had to do with SQL… a lot simply had to do with life. Some of them are still simmering, but all of them will end up in a story sooner or later.

Again – thanks to…

…all who attended the presentation (Jason had said I couldn’t start until he was there – I had the pre-presentation butterflies and was quietly hoping for him to be late – but he was there in the front row when I got there – so there was no backing out at all :-) –

…to Avanade for giving me the time off to go do this, and

…to our wonderful friend for sharing her lovely home and hospitality with us, and last but most certainly not least

…to my family for their patience and faith in me as I worked through it.

There are many more lessons out there to be learned, and as I find them, I’ll do my best to share them.

Oh, here’s just one:

You know how Bugs Bunny always says, “I knew I should have taken a left turn at Albuquerque!“?

I now know why he couldn’t do that.  🙂

Take care out there, folks…

Tom

 
1 Comment

Posted by on March 2, 2015 in Uncategorized

 

Tags: , , , , , , , ,

T-SQL Tuesday #60– Something New Learned


 

So – as I write this, it’s the Monday evening after Summit

Emotions are still fresh, but my mind still trying to double-clutch its way back to thinking about how to apply all the things I’ve learned and I’ve been struggling to find words to answer the question asked for this one, but I think it was narrowed down to one tweet that we’ll get to in a moment.

Like many of you, while I work around a lot of people, and work with people, most of the work I do is fairly solitary. I don’t often get a chance to learn with or just relax and chat with folks outside of work who are fighting the same battles, have similar goals, and are struggling with similar  issues.

I’d been working quite a few overtime hours, plus getting ready for my first presentation in a long time for SQLSaturday 337, so I was a little frazzled coming into Summit14, and was, frankly, looking forward to what I’ve taken to calling “Summer camp for Geeks”.

Take the best thing you can remember from any summer camp you’ve ever been to. (For me, regardless of the camp, it’s the camaraderie – so let’s go with that).

Take that camaraderie – and add to it the concept of a support group. Oh, for those who haven’t been part of a support group, let’s just go to our old friend Wikipedia, which states, in part:

In a support group, members provide each other with various types of help, usually nonprofessional and nonmaterial, for a particular shared, usually burdensome, characteristic. Members with the same issues can come together for sharing coping strategies, to feel more empowered and for a sense of community. The help may take the form of providing and evaluating relevant information, relating personal experiences, listening to and accepting others’ experiences, providing sympathetic understanding and establishing social networks.

Wait – let’s look at those last two sentences again one at a time…

“Members with the same issues can come together for sharing coping strategies, to feel more empowered and for a sense of community.”

Uh – yeah – there’s 5,000 of us from all over the planet, most of us working in pretty solitary kinds of positions, where it’s often hard to talk about the issues we’re facing, often because we’re the only ones in the company facing them. Interestingly, we’re often introverts, which means at some level, we’re okay with the whole “working by ourselves” thing. But it is nice to know you’re not the only one out there trying to invent this particular wheel, or shove this particular boulder up a hill… Again… (see Sisyphus). That brings us to the second of the two sentences:

“The help may take the form of providing and evaluating relevant information, relating personal experiences, listening to and accepting others’ experiences, providing sympathetic understanding and establishing social networks.”

Just read that again… I know, it’s likely the third time… Right – then look at the picture of the community zone below from the tweet by Neil Hambly (B|T)

I was just stunned at the simplicity of Neil’s comment, “This what #sqlhelp is like in the flesh Many folks just in the community zone waiting to help others”

Just from this shot, I see Kevin Kline (B|T) chatting with someone on the floor on the left. Wendy Pastrick (B|T) is chatting with someone about dead center. I think that might be Grant Fritchey (B|T) there in one of the blue shirts in the background. I know for sure there was at least one SQUEEEE! (B|T) there. 😉

I know there are other folks I didn’t get a chance to meet or can’t recognize in this shot – but I was in the same spot that Neil was when he took that picture and saw the same thing.

And heard the same thing.

And it was people simply wanting to help others who were going down the same path. People would come to that blue carpet and be welcomed in. They’d reach out for help, and there was someone in there, reaching back. Understand, there were folks just starting their careers in there mixed right in with folks who were pillars of the SQL community, there were beginners, MVP’s, and MCM’s, and all were reaching out to each other to help.

That help took all forms. Sometimes, someone just needed to talk something out. Other times, it was a deep conversation. Often there was laughter, and what became known as #sqlhugs as those of us who’d emailed or tweeted with others for years finally got to meet them face to face. In the shot, you see people talking with each other, you see people on their laptops while on support calls during the conference itself (it was obvious on some, you could hear one side of the conversation, “…Okay, now click ‘next’…”)

And for some, just like Neil said, it was #sqlhelp in the flesh.

If you’re not familiar with the #sqlhelp hashtag – click on that one on the left there – it’ll take you to the post Brent Ozar (B|T) put out on his blog to explain how it works, with the opening line: “You’ve got questions, we’ve got answers.”

And at summit – sqlhelp wasn’t just something we typed.

We got to see some of the people behind it (that’s be all of us)

We got to meet them.

And we got to experience that camaraderie of summer camp and the warm embrace of a support group.

We got to hear the laughter behind an LOL as someone got a joke, or got to actually see, talk to, and listen to the folks behind it.

And over time, what became clear to me is that while I’ve used #sqlhelp in the past and monitor it in the present, #sqlhelp isn’t just some search engine that spits out answers like ones and zeroes. . It’s real people, with real experience (and real questions), again, some just starting out, some, like I said, pillars of the community, but who all take or make the time to get an answer for you, and then, often, check back with you to make sure you got it.

#sqlhelp is folks who care, folks who’ve reached forward to others in their careers and are now reaching back to help those who need it.

That’s one thing I learned at Summit this year. Not just in my head, but in my heart.

And in case no one’s said it out loud, it’s appreciated.

 
5 Comments

Posted by on November 11, 2014 in Uncategorized

 

Tags: , , , , ,

Coffee Grinders, Wisdom, and PASS


I’ve been at the SQL Pass Summit in Seattle this week, and I can honestly feel my head expanding from all the incredible information and knowledge being shoved in there by so many people who have worked so hard to put their presentations together, to share their experiences, their knowledge, and their mistakes.

And the other morning, as I was idly waiting for the hot water dripping through the coffee maker to become something useful, it got me thinking, about how often that has happened in my life, someone sharing hard earned wisdom that saved me from having to learn from my own mistakes.

I kept watching the coffee maker for a bit, then started to see the pieces come together.

See, not only is the PASS Summit in Seattle, I live in Seattle.  Coffee is more than just a ritual here, it is one of the basic food groups.  It is as necessary as, say, air, and that morning, I was about to clean out the coffee grinder when I heard my Oma (German for Grandma) stop me.

The thing is, she’s been gone from this earth, to live on in our memories, for over 30 years.  Why would she be stopping me? –

And then I remembered.

When I was little, she had a very similar coffee grinder (a Braun 220 volt one that you could wrap the cord around), and she ground the coffee the same way I do.  Fill the grinder about halfway with coffee beans, put the lid on, mash down on the button, wait until the rattle turns into a hiss, and you’re done.  Then – if it was her, she’d unplug the coffee grinder, then spoon out a measured amount of coffee into the coffee pot, if it’s me I just dump it into the filter I’ve put into the coffee maker.

But it’s what she said, in that southern German Swabian dialect I grew up hearing, as she unplugged it that stopped me.  “Also domme Leut’ däen d’r draht in d’r Steckdose bleiba lassa.” (roughly translated: „Dumb people leave it plugged in.”) – and I just took it to be a bit of gray haired wisdom that I didn’t quite understand until years later.

See, that wisdom was pretty simple: Don’t do stupid things.

In fact, it seems that her advice could have been used about 40 years later… See, it seems that a small coffee company here in Seattle had a coffee grinder made that really wasn’t too different from the one Oma had way back when.  It turns out that several people (some of them smart in some ways) managed to actually injure themselves by getting (or putting) their fingers into this little machine.  One that’s designed to take small things and make them into much smaller things.  Apparently the coffee grinders turned on while they were trying to clean them, and that advice about unplugging… a machine… that’s (let me repeat this) designed to take small objects and make them smaller… to keep it from turning on could have been quite useful. But my Oma wasn’t around to give that kind of advice, and as a result, the advice wasn’t heeded, and the company recalled over half a million of the things.

And that wouldn’t have happened if they’d been unplugged.

After I’d thought about Oma, (having unplugged the coffee grinder and put it away) I had another thought, this time about something that happened with Opa (German for Grampa) a few weeks later, having nothing at all to do with coffee.

See, growing up in Germany when I did, the towns and cities still had more than just a touch of a medieval feel to them, what with the cobblestone streets, and some towns still snug inside the walls that were designed to keep the inhabitants in, and any attackers out.  It had been this way for centuries. People were inside the walls, all the agriculture was on the outside, and one day, I went with Opa to one of the gardens that had been in the family for generations, where there were plum trees and potatoes, cabbage and cauliflower, beans and broccoli.

But to get to the garden, we had to walk through a long, narrow orchard, and the only thing Opa had to cut the grass that grew there among the trees with was a scythe, so it didn’t get cut often, and was about waist high on me.  I didn’t really care, and was having fun running through it when I noticed Opa was gone.  He was short but not that short, so I hunted a bit and found he’d crossed over the fence into the neighbor’s orchard, which, as I recall, had horses on it occasionally, and the grass there was much shorter as a result.  It really surprised me, as it was some work to get over the fence and he was over 80 years old at the time, so it was counterintuitive to even think of him going over the fence to walk down along the fence only to have to come back, and I was young and didn’t care about doing something as simple as walking efficiently (on the other side, where experience had told Opa that it was easier).  I just cared about doing it, even if it wasn’t efficient.  In fact, I asked him why he was over there, and he invited me to come over and join him, saying, in that Swabian dialect, “S’isch leichter zom laufa auf dära seite.”  (“It’s easier to walk on this side.”)

I didn’t go over there, being full of spit and vinegar (or the German equivalent) , as it were, and he didn’t push it and let me run like a young maniac through the tall grass on one side of the fence while he walked sedately over the short grass on the other side.

And it got me thinking…

Again.

There was something about that gray hair of theirs.

It symbolized wisdom.

Oma didn’t stick her fingers into the coffee grinder without unplugging it first.

That kept her from losing bits and pieces of her fingers in the grinder.

Opa did walk on the right side, where it was easier.

That was where he didn’t have to waste energy he needed for working in the garden on something as mundane as getting there.

So those were some pretty non-technical examples, but you can easily translate those into all the technical things you do, whether that’s making sure you don’t do the IT equivalent of sticking your fingers in a coffee grinder, or making sure you do the IT equivalent of taking the time to walk down the path on the other side of the fence.

While you’re here – take the opportunities you have and learn from those who have gone before, and who have done stupid things (and learned how to fix them) because you won’t live long enough to make all the stupid mistakes there are to make.

Take the opportunities, some of which you only have here at PASS.  Learn from those who have gone before, and who have done wise things, because you won’t live long enough to learn all the wise things yourself.

Look around you at the PASS Summit, both when you’re between sessions and especially at lunch.

See, most people here with some gray hair (figuratively or literally) didn’t get to be that way in this industry by being stupid.  Pay attention to the things they do. The things they do are often the result of hard won wisdom.

You’ve heard that wisdom (or, good judgment) comes from experience.

Of course, experience often comes from bad judgment.

So when you see a person who looks like they’ve been around awhile here at PASS, pay attention, not only to what they do, but also pay attention to the things they don’t do.

And don’t be afraid to ask them why.

You might learn something.

Oh, and if you’re still in the industry when you find that first little gray hair of your own sprouting, and you see a wide eyed up and coming DBA looking at you as if you’re the fount of all knowledge, shyly asking some advice on something, just to say they heard it from you, remember what it was like when you were thirsty for all that knowledge, and treat that request with  the respect and humility it deserves, because once that happens, it’s definitely time to give back to the community that helped make you who you are.

Take care –

Tom

 
7 Comments

Posted by on November 7, 2012 in Uncategorized

 

Tags: , , , , ,