Weekend spamtacular -- what the heck happened and how we're fixing it

Oh.  My.  Gosh.  This weekend sucked.  No doubt about it.  But we've beaten things back and we have a plan for making things better still.  I'm going to tell you all about that in a minute.

But first, I owe everyone who was affected a massive apology.  All the success that has come to MyBlogLog has been because of your passion for our service and I hate when something happens that causes that love to diminish.  We left a hole in the code and a lot of people received a bunch of irrelevant email notifications because of it.  Our bad.

In order to describe what happened this weekend, it's worth laying out a few pieces of context.  First, in the States it was a three day weekend, so everyone was basking in the thought of staying offline for a few days and coming back recharged.  Todd and John were back in Orlando at a wedding and Steve, who just moved out here, is busy looking for a place to live.  And I'm splitting my free time between unpacking and giving my wife time off from watching our 16-month-old.  No one was looking online.

Saturday evening, a member discovered an exploit where you could send someone a request to join their community as a co-author and then automatically approve the request.  In other words, someone (dare I call them a jackass) could force you to be a co-author of their community.  I have no idea why they would do this, other than a negligible bump in marketing, but who ever said jackasses made sense?

Early Sunday evening we were alerted to the problem.  Unfortunately, we didn't grok the problem initially.  We just thought that someone had used a script to send out thousands of requests for co-authors, which we promptly shuttered.  It wasn't until almost midnight, when Steve had gotten back home and Todd had just landed from a cross-country flight, that we understood the bigger exploit, which we also promptly shut down.  But it was too late by then, because the flood of emails had already struck.

This grief probably belongs in some frickin' griefing hall of fame (with jackasses on both sides of the entrance, mind you):

1. Send out thousands of emails to random people requesting that they co-author your community
2. Force-join them all as co-authors
3. Someone gets upset about being force-joined and leaves an angry message on the community, and EVERY single person gets an email alert that there's a message waiting for them (because they're all co-authors)
4. Now you have dozens of angry people, all leaving angry messages on the community page, resulting in DOZENS of emails alerts being sent out to each victim
5. And so on...

If you were one of the people that received a couple dozen email alerts about new messages, I am really sorry.  It has all been fixed and no one should be able to force join anyone else again.  We've rolled back all the new co-authors since Friday night so no one should find themselves co-author of something random.  And while we can't pull all of those emails back into the server, we've deactivated them, so even if you mistakenly click on the approval link, you still won't become a co-author.

But we're not stopping there.  As members who read this blog regularly know, we've been trying to figure out how to reduce the "friend" and "join" and "message" spam for weeks now.  Pretty much since last November.  What's tough is that a lot of the behaviors that tech-savvy members find infuriating (such as people sending messages to random recipients asking them to check out their community) are actually enjoyed by casual members.  So we have to find a balance.

The team has spent the bulk of their holiday working out a plan of action for the next couple of weeks based upon feedback from a lot of users.  I invite you to comment on the plan below and let us know if you think we've gone too far anywhere and if we've missed something that you think is vital.

MyBlogLog's Six Point Plan to Spiritual Nirvana:

1) We're going to post an official Terms of Service (ToS) and hold people accountable.  It's hard kicking people's asses for breaking the rules when the rules aren't posted anywhere.  That will change.  Things like blatant advertising in profiles will not be tolerated.

2) By default, you now see only message from your own contacts. You'll be able to click a radio button to see messages from everyone else.  Further, you'll only receive an email alert when a contact leaves you a message.  Lastly, public views of your profile will reflect your message view setting, so other people viewing your profile won't see random requests to visit their community or site.

3) We will include the text of the comment and associated controls (delete,reply, etc) in the alert email.  You won't have to go to MyBlogLog to manage comments on your profile or community page any more.

4) We will limit users to only five requests for co-authors a day.  If you want to request more co-authors, come back tomorrow.

5) We will limit users to join 15 communities and add 15 contacts during any day.  The others will still be here tomorrow.

6) After the first five are complete, we will set up a comment approval system where community members can automatically post messages and everyone else's comments gets queued for approved (a la Typepad comments).

I'll be the first to admit it's not perfect.  Some of it feels a little arbitrary (15 joins per day) but it's the best that we've got for now.  Of course, we'll continue to listen to feedback after these new measures are deployed and if something is too strict or too lenient, we'll make more changes.

Here's hoping the next three-day weekend is nothing but pleasant messages and happy surfers.

Eric

Comments

[qureyoon]

Look at the bright side, those spammer are making MBL more and more perfect in any shape ;) so, keep 'em comin' :P

You guyz sure are quick on the respond ^^

Great move here ;)

[JohnC]

lol
growin' pains....ain't it kewl? Congrats on the hoop jumpin', folks.

[Tamar Weinberg]

Hey Eric and Scott and everyone else - your efforts are great, thanks. Thank you for promptly putting up a system to work around these matters.

I think that #2 and #6 work to address the biggest issues of the present and I look forward to seeing their implementation.

Thanks, guys!

[Darren]

great stuff - thanks for responding!

[rxbbx]

Dont feel guilty with limiting things when its nescessary. Thnx for your efforts.

[Barbara]

I got the first email you're speaking of, but as far as I know I wasn't automagically added. I didn't get any follow on emails after that one. I also did not click on the link in the email to become a co-author.

[markus941]

Sounds like a case of the Mondays ;)

Glad to see you being upfront and putting out the fires with transparency and all. Kudos.

One other thing I'd suggest addressing is the "avatar issue" - the animated gifs can be pretty distracting (and hefty in file size too). Some kind of author-control of animation/no-animation and caps on file avatar sizes would be awesome.

[Bradford Knowlton]

I spent sometime over the weekend looking at what happened. And realized MBL puts a ton of credibility into cookies.

My recommendation for the adjustment to the Co-Author request is the following:
Currently there is a two variables, the SID and MID.

If you replaced them with a hashed value, most of the problems this weekend would have been avoided. If you take the MID multiply by the SID, you get a value. Send the product, and the SID in the e-mail link. Knowing the MID from the cookie value, you can decode the product, and validate the e-mail link. Sounds like a solid plan. You could even send the first 1/2 of the product, or some checksum value.

The other issue with your cookies, is that simple adjustment to the cookie values allows people to spoof their appearance, and show up as "other people" when browsing the web.

[BO18]

Thanks!
I hope it helps.

Mybloglog is too nice to be devastated by spammers.

[HMTKSteve]

Will you be adding the ability to set one author as the owner so they can remove co-authors if needed?

[Lord Matt]

By making the limit so low you increase the value of the join / add. Because it is a vote of confidence rather than spam. So if I get new community members then I have won those guys with my fantastic ninja-pirate-god-like blogging! Once it settles down it should add value.

[Shuttlecock]

Great news. I love the idea of limited communities people can join and friends they can add.

[Louis Gray]

And here I thought I had a real opportunity to edit BlogMemes Belgium! It must have been due to my tremendous insight on all things Belgium!...

We weren't terribly perturbed by the spamming, but it's good to see the response. Hopefully, you can avoid becoming the next eBay or PayPal, in that I'll never trust an e-mail purportedly from those folks.

[Robert]

Jackass indeed.

[Loretta]

Glad it's all getting worked out :) Thanks for taking care of us.

[Kate]

Thanks for working on the problem. I'm particularly pleasd to see number 5.

[rendy]

i get spam too

i didnt know about blog memes community
http://www.mybloglog.com/buzz/community/BlogMemes_Belgium/

but i get more than 50 messages to my inbox from this community..

thanks to eric that have fixed the function, hope your mybloglog.com function works properly and get more feature :)

[Charbarred]

Nice and fast response. A usual you guys are providing a great service.
I see that Steve made a MyBlogLog wishlist which I completely agree with. Just thought I'd bring it to your attention:
http://www.hmtk.com/archives/my-mybloglog-wish-list.html

[John Egan]

Only joined within last 24 hours. Already see value to community concept, especially when organizers quickly resolve issues. That's real value added.

[Easton Ellsworth]

You guys'll get this figured out. Great work so far and keep building this community.

[Giovanni Gallucci]

Eric,

I am amazed and pleasantly suprised and the quickness of your team's response and the thoroughness of the explaination to the community. You folks are good people. Thanks for the dedication to the cause!

-giovanni

[Joe Duck]

Outstanding rapid response to the problem Eric. I'm really impressed with how you guys are handling the big transitions and the glitches like this. Now is the time to kill off abusive stuff and you are doing it.

[Stephanie]

Well, that explains that email. I wondered what the guy was up to since it made no sense to be asking me to coauthor his blog. Real sneaky. Glad you fixed it.

[Missy Caulk]

Yea, I was sent a multitude of emails;but thankfully I was able to figure out how to stop it. After I started getting the emails I thought what an idiot. Oh well a great lesson

[claude]

Blogmemes is not responsible for the hack which occurred this weekend via the Mybloglog Web service.
The Mybloglog account of one of the network's members was pirated without his knowledge.
We do not yet know why or who might be responsible.
It is not in line with the network's code of ethics nor in its interest to proceed in this manner and serves only to discredit our community.
We are currently suffering many spam attacks on our Web sites, which we are combatting as much as possible.
We thank the mybloglog team for having now corrected this problem.

We are currently trying to answer all those who have written to us, to explain the situation to them.

Claude
Co-founder of the blogmemes network