“Cowboy coding” is the phrase we use around here to describe the practice of changing code directly on production servers. It’s not really a best practice, and we generally discourage it, but every once in a while it becomes necessary…
Development and deployment here normally consist of a disciplined process that includes code reviews in development, a staging server for business owner signoff and QA, and finally a push to production. Any change to the code requires going through all these steps again. It’s time-tested and works well, ensuring that we push out quality code and avoid errors.
That said, it takes time, and time costs money. Our clients are looking for us to be responsive; when they want the headline changed to a different shade of green on their website (just after launch, of course,) they don’t want to hear it’s going to take two days. In those cases, it makes sense to jump onto the servers and make that one line CSS change–and make everyone happy immediately.

Even when it makes sense, cowboy coding is not something we take lightly. Anyone editing code on a production server is required to wear this pink sombrero for the duration of the work.

Having the pink sombrero on acts as a strong check and balance. Inevitably, the act of donning this flamboyant headwear causes a crowd to form. Lively discussion ensues:
“What do you have to edit? Do we really have to do this? Are we sure there will be no side effects?”
And it works. Purists might tell you that you should NEVER edit code in production, and that makes sense in a lot of environments. But we build marketing websites, where speed and responsiveness count as much or more than perfect reliability.
For the record, the last time this happened was a last-minute change nine days ago — after we went live — for our own website, and the change came from me.
I’m the worst client.
If you enjoyed this post, I’d appreciate a vote to bring Pink Sombrero to SXSW. For more Pink Sombrero, check out Part 2 Avoiding the Pink Sombrero: Potential Failure Modes of Automated Deployment.
And here I thought M@ wore the pink sombrero just because it accentuated his eyes. Now I feel his pain.
This doesn’t work for people who don’t care about silly hats.
I like this idea
Certainly seems effective and it’s nice to see companies that realize that sometimes “Best practice is not practical practice”
http://en.wikipedia.org/wiki/Cowboy_coding
anonymous fool is anonymous
What a great hat! I would be the worst employee, because I would want to wear it all the time.
Before the CVS era there was the “rubber duck”. whoever had the rubber duck was editing code in the shared code repository.
Careful. I hear valve has the patent on silly hats
We just yell:
“We’ll do it LIVE”
Taken from 1:00m into this video: http://www.youtube.com/watch?v=2tJjNVVwRCY
This is great. We have a “fail sombrero” at work that you have to wear in the case of any large defect.
Front page of reddit — congratulations. I do this too but I don’t wear a hat.
Something like this…
The client should have to wear the hat, not the developer.
what about version control? doing stuff on the server immediately puts items out of sync no?
dev->commit->update single file on prod->run away
you also can use blame command then
There should be a variety of processes that require some sort of sign-off for everything from 1 hour changes to weekly/monthly.
Obviously streamlined sign-offs for hourly. Much less so for the latter.
Heh … this is awful but we call those “handjobs”, because you’re doing the job by hand on the server instead of using version control & configuration management & all the regular process.
i’ve been begging for test/dev servers for years. no one understands how wrong it is to only have a production server…ugh…we’re always wearing the pink sombrero
I pushed a little bit the concept further.
I frequently simulate service call, add behaviour to our services through groovy aspect (add more logging, add measure point,…) through groovy console built-in in our apps.
(inspired by jenkins system scripts https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+System+Groovy+script)
At Facebook we get bright red clown noses whenever we case a service degradation or do something just plain clowny. We also elect a mayor of clowntown for serious issues. Fortunately we have no way to edit production code, code reviews and testing are mandatory and code is only pushed to production through a structured push process (which can be accelerated to an hour to fix serious live bugs).
If I had to wear that pink sombrero, my next step for the day would be to document the incident and then talk to my attorney after work because it’s a discrimination case that might net six figures or more in a settlement. Why work when your company is so stupid and unprofessional as to allow acts of hazing against employees?
Jake… you’re either a litigious dickhead or a boring dickhead.
If your only choice is 2 days of QA or immediate editing of production code, your process is wrong. Direct production code changes should only be done in cases where the production site is not functional at all (which is almost exclusively caused by direct editing of production code for less important reasons).
Why not modify your process so in these “emergencies” you can still test your change on a staging site and then use version control to update your live site? It only adds 2-3 minutes and eliminates a lot of risk.
Love the term Josh. I’m going to have to borrow that because I recently experienced this with a new system implementation and had a fit when I found out about it. “Cowboy coding” in an accounting system is a major SOX violation! Hope all is well with the B&J gang!
It’s not necessary to become a cowboy to deploy changes to production in a very short time. Flickr, Facebook, and many other companies deploy changes to production 10 times a day in a fully disciplined way. This, of course, is called continuous delivery.
many years ago, the group I was in had a large orange traffic cone. If you broke the build, the cone moved to your office and stayed there until someone else broke the build….
of course for this not to be demeaning there needs to be a very friendly atmosphere around the company
and I really liked it hehe
I didn’t like the orange traffic cone one because I think it would bring a negative competition(as for hoping someone else would brake the build so you wouldnt be “it”, and enjoying it…)
Pingback: Cowboy Coding, No Sombrero » the Void
To clarify for those people who are insensed by this idea and have their lawyers on speed dial, this is not a management policy. This is something we (developers) came up with on our own. It’s Matt’s personal sombrero and I’m probably to blame for the misuse (according to Wikipedia) of the term “Cowboy Code.” Nobody likes Cowboy Coding, but in Marketing things can become… less controlled than we analytical tech types like. So we came up with the Pink Sombrero ourselves to keep our mood up and our outlook positive.
Yeah, if someone was making us don a Sombrero of Shame, well that would not fly. But you cannot be grumpy, even about a live code change, grinning at your coworkers, looking like the hariest princess of the rodeo.
As the handsome model of this pink sombrero, I can tell you that wearing the hat is not an act of shame, and that we aren’t “forced” to wear it (as Ben just pointed out). We, the Programmers of Babcock & Jenkins, in order to form a more perfect process, establish Visibility, ensure Accountable Modifications and secure the Blessings of Project Managers, do don and establish this Pink Sombrero. It was an accountable practice that we web developers & programmers took upon ourselves to promote discussion and to try to prevent Black Swans from being introduced into untested production changes on client(and internal)-facing content.
It’s a practice done for our own peace of mind, not something to invite shame and humiliation.
We used to have a support helmet. When a customer issue reached engineering, and you had to fix it, you wore the helmet.
No, I’m pretty sure I’m the worst client.
Pingback: WorkSmart Labs Blog » Blog Archive » Sleuthing Git using Git, or, Regression testing done backwards
Pingback: Avoiding the Pink Sombrero: Potential Failure Modes of Automated Deployment | Babcock & Jenkins
Pingback: Around the web | alexking.org
“Fortunately we have no way to edit production code” — this might be the best way to protect yourself.
“Crown of thorns” might be more appropriate, and the person that requested it should also have to wear it, all the way up the chain…
This is old news.
Sombrero’s were already cool almost 30 years ago…
http://www.baronhats.com/images/thing1.jpg
Pingback: M3talinks For 08/03/2011
I am a bit to lazy to read through all the posts, so forgive me if I am repeating this question.
Where can I obtain a pink sombrero like the one pictured? I will be taking up this practice.
It was given to me by an ex co-worker long ago in exchange for a plastic viking helmet. Its origins are unknown.
Pingback: Web Design Weekly #4 | Web Design Weekly
Pingback: Fun?! with Subversion and WordPress | WerdsWords
Pingback: When Reddit Attacks! Traffic Profile of a Front Page Post | Babcock & Jenkins
Pingback: Link del día: Programando con el sombrero rosa | Alpha's Manifesto
Pingback: 牛仔式编程和粉红色的大檐帽 - 博客 - 伯乐在线
I love the idea!!! Thanks for sharing!
I know its already been said that ideally you wouldn’t have to do this. It really is true. If you had great CI and a load of testing then you could make the change in your production branch, all tests run green, then deploy the sucker. Getting to quick deploys of well tested code should be the goal, not the hat of shame.
Pingback: Cowboy coding | | base42.nlbase42.nl
Pingback: Konstantin is Cowboy Coding - Konstantin Kovshenin
Pingback: Finely Tuned Consultant: Gary Jones | WordPress Hosting by WP Engine
Pingback: Quick Tip - Work on Local Files with Diet Coda
Where can I buy a pink sombrero?!
Pingback: Dev to Production: How to Deploy WordPress Code via Version Control
Pingback: Puppetcast Episode 5 | Puppet Labs
Um… I just have to wonder why the pink sombrero. Since neither pink, or sombreros, have anything particular to do with “cowboys” at all.
What does being a gay Mexican have to do with being a cowboy?
Why not have him (her) wear a 10-gallon hat, pointy shoes, and spurs? Maybe chaps, even. Now THAT would be cowboy.
Clarification: I meant nothing either racist or sexist, but I kind of think the pink sombrero does.
Two red flags.
1) Getting a small change through QA and signed off takes far too long
2) You change things directly on prod
Surely a small CSS change would be handled via content management? The fact these things are hard coded is shocking.
There really is no reason to be modifying things on production. You need some serious process improvement.
Pingback: Simpleweb
@Craig when you pump out sites and are understaffed, not all projects even have a CMS!
The first thing you need to do before anything else is to get yourself a domain name. A domain name is the name you want to give to your website. For example, the domain name of the website you’re reading is “thesitewizard.com”. To get a domain name, you have to pay an annual fee to a registrar for the right to use that name. Getting a name does not get you a website or anything like that. It’s just a name. It’s sort of like registering a business name in the brick-and-mortar world; having that business name does not mean that you also have the shop premises to go with the name.:
Have a look at the best and newest content at our new website
<".http://www.beautyfashiondigest.com/keratin-treatment-reviews/
Pingback: Announcing User Portal v 2.0 and Ability to Push from Staging to Production | WordPress Hosting by @WPEngine