“Cowboy coding” is the phrase we use around here to describe the practice of changing code directly on production servers. It’s not really a best practice, and we generally discourage it, but every once in a while it becomes necessary…

Development and deployment here normally consist of a disciplined process that includes code reviews in development, a staging server for business owner signoff and QA, and finally a push to production. Any change to the code requires going through all these steps again. It’s time-tested and works well, ensuring that we push out quality code and avoid errors.

That said, it takes time, and time costs money. Our clients are looking for us to be responsive; when they want the headline changed to a different shade of green on their website (just after launch, of course,) they don’t want to hear it’s going to take two days. In those cases, it makes sense to jump onto the servers and make that one line CSS change–and make everyone happy immediately.

Pink Sombrero

Even when it makes sense, cowboy coding is not something we take lightly. Anyone editing code on a production server is required to wear this pink sombrero for the duration of the work.

Pink Sombrero

Having the pink sombrero on acts as a strong check and balance. Inevitably, the act of donning this flamboyant headwear causes a crowd to form. Lively discussion ensues:

“What do you have to edit? Do we really have to do this? Are we sure there will be no side effects?”

And it works. Purists might tell you that you should NEVER edit code in production, and that makes sense in a lot of environments. But we build marketing websites, where speed and responsiveness count as much or more than perfect reliability.

For the record, the last time this happened was a last-minute change nine days ago — after we went live — for our own website, and the change came from me.

I’m the worst client.

If you enjoyed this post, I’d appreciate a vote to bring Pink Sombrero to SXSW. For more Pink Sombrero, check out Part 2 Avoiding the Pink Sombrero: Potential Failure Modes of Automated Deployment.




avatar
60
Comments

Comments.

    Noel Ponthieux
    June 29, 2011 at 6:55 pm

    And here I thought M@ wore the pink sombrero just because it accentuated his eyes. Now I feel his pain.

    no
    July 5, 2011 at 11:45 pm

    This doesn’t work for people who don’t care about silly hats.

    Seth
    July 6, 2011 at 12:24 am

    I like this idea ;)
    Certainly seems effective and it’s nice to see companies that realize that sometimes “Best practice is not practical practice” :)

    Dejan
    July 6, 2011 at 12:26 am

    http://en.wikipedia.org/wiki/Cowboy_coding

    anonymous fool
    July 6, 2011 at 1:44 am

    anonymous fool is anonymous

    Glen
    July 6, 2011 at 1:58 am

    What a great hat! I would be the worst employee, because I would want to wear it all the time.

    July 6, 2011 at 2:00 am

    Before the CVS era there was the “rubber duck”. whoever had the rubber duck was editing code in the shared code repository.

    tWoolie
    July 6, 2011 at 2:10 am

    Careful. I hear valve has the patent on silly hats

    July 6, 2011 at 3:13 am

    We just yell:

    “We’ll do it LIVE”

    Taken from 1:00m into this video: http://www.youtube.com/watch?v=2tJjNVVwRCY

    July 6, 2011 at 6:01 am

    This is great. We have a “fail sombrero” at work that you have to wear in the case of any large defect.

    July 6, 2011 at 6:16 am

    Front page of reddit — congratulations. I do this too but I don’t wear a hat.

    July 6, 2011 at 7:22 am

    Something like this…

    Betty
    July 6, 2011 at 7:28 am

    The client should have to wear the hat, not the developer.

    July 6, 2011 at 8:23 am

    what about version control? doing stuff on the server immediately puts items out of sync no?

    dev->commit->update single file on prod->run away

    you also can use blame command then :)

    Anon
    July 6, 2011 at 9:16 am

    There should be a variety of processes that require some sort of sign-off for everything from 1 hour changes to weekly/monthly.

    Obviously streamlined sign-offs for hourly. Much less so for the latter.

    Abe
    July 6, 2011 at 10:58 am

    Heh … this is awful but we call those “handjobs”, because you’re doing the job by hand on the server instead of using version control & configuration management & all the regular process. :P

    Kyle
    July 6, 2011 at 12:56 pm

    i’ve been begging for test/dev servers for years. no one understands how wrong it is to only have a production server…ugh…we’re always wearing the pink sombrero

    sms
    July 6, 2011 at 1:45 pm

    I pushed a little bit the concept further.

    I frequently simulate service call, add behaviour to our services through groovy aspect (add more logging, add measure point,…) through groovy console built-in in our apps.
    (inspired by jenkins system scripts https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+System+Groovy+script)

    July 6, 2011 at 1:49 pm

    At Facebook we get bright red clown noses whenever we case a service degradation or do something just plain clowny. We also elect a mayor of clowntown for serious issues. Fortunately we have no way to edit production code, code reviews and testing are mandatory and code is only pushed to production through a structured push process (which can be accelerated to an hour to fix serious live bugs).

    Jake
    July 6, 2011 at 1:55 pm

    If I had to wear that pink sombrero, my next step for the day would be to document the incident and then talk to my attorney after work because it’s a discrimination case that might net six figures or more in a settlement. Why work when your company is so stupid and unprofessional as to allow acts of hazing against employees?

    Anon
    July 6, 2011 at 2:52 pm

    Jake… you’re either a litigious dickhead or a boring dickhead.

    Jon
    July 6, 2011 at 3:15 pm

    If your only choice is 2 days of QA or immediate editing of production code, your process is wrong. Direct production code changes should only be done in cases where the production site is not functional at all (which is almost exclusively caused by direct editing of production code for less important reasons).

    Why not modify your process so in these “emergencies” you can still test your change on a staging site and then use version control to update your live site? It only adds 2-3 minutes and eliminates a lot of risk.

    Gina CPA
    July 7, 2011 at 2:34 am

    Love the term Josh. I’m going to have to borrow that because I recently experienced this with a new system implementation and had a fit when I found out about it. “Cowboy coding” in an accounting system is a major SOX violation! Hope all is well with the B&J gang!

    July 7, 2011 at 12:35 pm

    when they want the headline changed to a different shade of green on their website (just after launch, of course,) they don’t want to hear it’s going to take two days.

    It’s not necessary to become a cowboy to deploy changes to production in a very short time. Flickr, Facebook, and many other companies deploy changes to production 10 times a day in a fully disciplined way. This, of course, is called continuous delivery.

    Bill
    July 7, 2011 at 3:17 pm

    many years ago, the group I was in had a large orange traffic cone. If you broke the build, the cone moved to your office and stayed there until someone else broke the build….

    Breno
    July 8, 2011 at 4:37 pm

    of course for this not to be demeaning there needs to be a very friendly atmosphere around the company ;)

    and I really liked it hehe

    I didn’t like the orange traffic cone one because I think it would bring a negative competition(as for hoping someone else would brake the build so you wouldnt be “it”, and enjoying it…)

  1. Pingback: Cowboy Coding, No Sombrero » the Void

  2. July 9, 2011 at 3:41 am

    To clarify for those people who are insensed by this idea and have their lawyers on speed dial, this is not a management policy. This is something we (developers) came up with on our own. It’s Matt’s personal sombrero and I’m probably to blame for the misuse (according to Wikipedia) of the term “Cowboy Code.” Nobody likes Cowboy Coding, but in Marketing things can become… less controlled than we analytical tech types like. So we came up with the Pink Sombrero ourselves to keep our mood up and our outlook positive.

    Yeah, if someone was making us don a Sombrero of Shame, well that would not fly. But you cannot be grumpy, even about a live code change, grinning at your coworkers, looking like the hariest princess of the rodeo.

    m@
    July 9, 2011 at 10:55 pm

    As the handsome model of this pink sombrero, I can tell you that wearing the hat is not an act of shame, and that we aren’t “forced” to wear it (as Ben just pointed out). We, the Programmers of Babcock & Jenkins, in order to form a more perfect process, establish Visibility, ensure Accountable Modifications and secure the Blessings of Project Managers, do don and establish this Pink Sombrero. It was an accountable practice that we web developers & programmers took upon ourselves to promote discussion and to try to prevent Black Swans from being introduced into untested production changes on client(and internal)-facing content.

    It’s a practice done for our own peace of mind, not something to invite shame and humiliation.

    JC
    July 10, 2011 at 3:43 am

    We used to have a support helmet. When a customer issue reached engineering, and you had to fix it, you wore the helmet.

    July 12, 2011 at 10:53 pm

    No, I’m pretty sure I’m the worst client.

  3. Pingback: WorkSmart Labs Blog » Blog Archive » Sleuthing Git using Git, or, Regression testing done backwards

  4. Pingback: Avoiding the Pink Sombrero: Potential Failure Modes of Automated Deployment | Babcock & Jenkins

  5. Pingback: Around the web | alexking.org

  6. stucco
    July 25, 2011 at 6:24 pm

    “Fortunately we have no way to edit production code” — this might be the best way to protect yourself.

    “Crown of thorns” might be more appropriate, and the person that requested it should also have to wear it, all the way up the chain…

    kr
    August 1, 2011 at 6:40 pm

    This is old news.

    Sombrero’s were already cool almost 30 years ago…

    http://www.baronhats.com/images/thing1.jpg

  7. Pingback: M3talinks For 08/03/2011

  8. Earl
    August 5, 2011 at 3:30 am

    I am a bit to lazy to read through all the posts, so forgive me if I am repeating this question.

    Where can I obtain a pink sombrero like the one pictured? I will be taking up this practice.

      mattg
      September 7, 2011 at 6:03 pm

      It was given to me by an ex co-worker long ago in exchange for a plastic viking helmet. Its origins are unknown.

  9. Pingback: Web Design Weekly #4 | Web Design Weekly

  10. Pingback: Fun?! with Subversion and WordPress | WerdsWords

  11. Pingback: When Reddit Attacks! Traffic Profile of a Front Page Post | Babcock & Jenkins

  12. Pingback: Link del día: Programando con el sombrero rosa | Alpha's Manifesto

  13. Pingback: 牛仔式编程和粉红色的大檐帽 - 博客 - 伯乐在线

  14. November 25, 2011 at 8:10 am

    I love the idea!!! Thanks for sharing!

    Bart
    December 15, 2011 at 4:55 pm

    I know its already been said that ideally you wouldn’t have to do this. It really is true. If you had great CI and a load of testing then you could make the change in your production branch, all tests run green, then deploy the sucker. Getting to quick deploys of well tested code should be the goal, not the hat of shame.

  15. Pingback: Cowboy coding | | base42.nlbase42.nl

  16. Pingback: Konstantin is Cowboy Coding - Konstantin Kovshenin

  17. Pingback: Finely Tuned Consultant: Gary Jones | WordPress Hosting by WP Engine

  18. Pingback: Quick Tip - Work on Local Files with Diet Coda

  19. June 12, 2012 at 11:08 am

    Where can I buy a pink sombrero?!

  20. Pingback: Dev to Production: How to Deploy WordPress Code via Version Control

  21. Pingback: Puppetcast Episode 5 | Puppet Labs

  22. Lonny Eachus
    September 22, 2012 at 4:11 am

    Um… I just have to wonder why the pink sombrero. Since neither pink, or sombreros, have anything particular to do with “cowboys” at all.

    What does being a gay Mexican have to do with being a cowboy?

    Why not have him (her) wear a 10-gallon hat, pointy shoes, and spurs? Maybe chaps, even. Now THAT would be cowboy.

    Lonny Eachus
    September 22, 2012 at 4:14 am

    Clarification: I meant nothing either racist or sexist, but I kind of think the pink sombrero does.

    Craig
    February 28, 2013 at 12:15 pm

    Two red flags.

    1) Getting a small change through QA and signed off takes far too long

    2) You change things directly on prod

    Surely a small CSS change would be handled via content management? The fact these things are hard coded is shocking.

    There really is no reason to be modifying things on production. You need some serious process improvement.

  23. Pingback: Simpleweb

  24. April 4, 2013 at 10:18 pm

    @Craig when you pump out sites and are understaffed, not all projects even have a CMS!

    Elliot Jardel
    April 8, 2013 at 7:57 am

    The first thing you need to do before anything else is to get yourself a domain name. A domain name is the name you want to give to your website. For example, the domain name of the website you’re reading is “thesitewizard.com”. To get a domain name, you have to pay an annual fee to a registrar for the right to use that name. Getting a name does not get you a website or anything like that. It’s just a name. It’s sort of like registering a business name in the brick-and-mortar world; having that business name does not mean that you also have the shop premises to go with the name.:

    Have a look at the best and newest content at our new website
    <".http://www.beautyfashiondigest.com/keratin-treatment-reviews/

  25. Pingback: Announcing User Portal v 2.0 and Ability to Push from Staging to Production | WordPress Hosting by @WPEngine

Add your own insightful remarks.

Your email address will not be published.

 Name

 Website (optional)




You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>