What happens when you break your site’s daily usage record… by 10x?

October 10, 2013

A retrospective on Fundfill’s biggest day

Yesterday was a bit of a crazy day! Since retweets of the fund for auditing TrueCrypt (fundfill.com/fund/TrueCryptAudited) began circulating and HackerNews featured the site, we’ve gotten roughly 35,000 views of the fund and more than 100,000 hits to our site. The most traffic we had had was 3,000 views in a day before, so we braced to see how the site would fare. I’ve read articles where the link to a site goes down because of a web server not being able to handle the load, so I braced for the worst. No need, as the site itself was able to display the pages with no speed issues at all! Yet, we started getting reports of users unable to pledge, and the number of pledges wasn’t moving at all. Users were registering, but nobody was donating? Looking at the site myself, I noticed every 10th page view would display “An error has occurred.” Hrmmmmmm.

We had tested the site vigorously leading up to this. Load tests had worked really well after we tuned up some issues 3 months ago, and our suite of UI tests, designed to test the user’s expected interaction with the site, passed every time we pushed out a new build. I had seen an error (regarding “unit of work”) about 1/100 times I was working on the site and I never could isolate the issue to reproduce it to fix it.

So, what happened?

Many users tried out our site for the first time yesterday. In the three hours after it started getting tweeted, there was a major problem in the code. Users who tried pledging were getting “An error has occurred” multiple times while trying to pledge, even though they could register. We identified the error and were able to fix it AND release a secure deploy within an hour after that. Unfortunately, there was also a one-time setting in the site I had changed to highlight the TrueCrypt fund. This “featured fund” setting was designed to expose the fund on the homepage. It had an adverse affect of temporarily changing the meta-data to the previous “featured fund” for a bounty on a killer during Bay-to-Breakers, a local race in San Francisco. So, users visiting the site saw a different fund than the one they were expecting. This snafu was caught within an hour of my causing it and fixed soon thereafter.

The other bug, for those technically inclined, was a race condition in a unit of work. Units of work are context boundaries that allow database changes to be collected and executed at one time to more efficiently separate code and access the database. There are two separate architectures in our code that create units of work. It turns out we had a race condition between these two pieces, causing the database to fire earlier than expected under heavy load. Once we were able to identify it, we organized the two pieces to perform database unit of work closing at the same time.

The new release not only smoothed out the site’s errors, but it also allowed users who had problems to finish their pledges. Contacting some of those users, we were able to get back some of the pledges we would have lost. Overall, we had over an hour of the Stephen Martin / Bay-to-Breakers fund being shown in place of the actual TrueCrypt audit fund and three hours of being unable to pledge. However, we’ve had zero interruptions in the last 16 hours, and we’ve tripled the previous days pledges to $1910. Given the current momentum, we expect another 2x to 4x increase in pledging by day’s end.

Lessons learned

For anyone out there with a website that needs evolve rapidly, invest in continuous deployment. Have tests that verify everything about your site and any bug that pops up, as well. Not just unit tests, but UI tests that can confirm that any user can perform all the actions your website requires. Once the root causes were identified, they were fixed within an hour – 20 minutes for the metadata chage, one hour for the pledging bug. Without the protections provided by these, we would have no confidence pushing out another build. Instead, we were confidently able to push out and provide a working site because we unit test and UI test every build before approving it.

Get feedback – One of the issues we faced was finding all the feedback actually going on. When something becomes popular very quickly, the internet has a way of transforming and creating communities to discuss problems – even if you don’t know they exist yet. Twitter is amazing at this by using it’s hashtags to create a grassroots communities out of nothing in a matter of hours. It wasn’t twitter, though, that had the most valuable feedback. I discovered an hour after they posted that HackerNews had started dissecting the fund. We were able to read and and gain valuable feedback about users’ experiences (and some much deserved criticism). Following a variety of terms on twitter and watching hackernews thereafter, we were able to see everything users were posting about the site and responding appropriately.

Communication – If a problem is behind-the-scenes and affects only one or two users, it’s best simply to fix the problem and contact the user who had the issue. In our case, everyone who visited the site knew that something was up, so not addressing it would have been a huge mistake. Green banners said “An error has occurred” and the site sometimes redirected to the homepage (the default behavior when the unit of work errors messed with the site). Reaching out to everyone and broadcasting our progress loudly was the best course to ensure everyone was aware of our issues. For the 5 users who added money but couldn’t pledge, I was able to directly contact them by phone, email, twitter and get their issues resolved. Overall, we got very positive reception from the users we were able to contact and in the forums where we addressed the issue. In business in general, I always advise people to be upfront and direct and honest. If you’re not honest, I won’t work with you, so I have to hold us to the same level of accountability.

In the end, I was really proud of the systems we put in place long ago that allowed us to make a quick recovery. Fundfill is designed specifically for bounties and rewards, so while there may be other sites for crowdfunding, they don’t necessarily cater to awarding money to the person who wins. Furthermore, Fundfill allows users to vote for the winner, based on the amount of money they donated. Did you donate your money but don’t feel the person claiming the prize actually did what they’re supposed to? You can reject their claim and demand the the right work for your bounty money. Plus, unlike the iPhone TouchId bounty that was hosted and operated via twitter and a website put together from scratch, we handle all the operational details for you – pledging, updating the money, and informing everyone via Twitter.

We’re still working out some operational details like what happens to the money if nobody is able to fulfill the fund. The fund’s creators will decide how long the money is kept in escrow and what to do with it should there be a failed bounty. If you’d like to discuss this directly with me, please find me on twitter at @joebalfantz or @fundfill. If you’re interested in donating, please check our site – there’s a link on the homepage.

Premature Celebration? Saints-49ers preseason

August 14, 2011

Watching a Saints’ game with an opposing team’s announcers was a pretty interesting scenario, highlighting the tremendous leaps-and-bounds the nation’s opinion of Black and Gold has leapt to. Overall, it was an encouraging game, but as I cautioned my mom back home – don’t look too deeply into it.

For starters, the Saints were a well-prepared team with continuity and consistency in too many ways – coaching and schemes, players, familiarity with teammates, and winning attitude – that the 49ers couldn’t compete with. The Niners are playing under a rookie coach who has had very little time to coach up his guys, given the long lockout shortening training camps NFL-wide, and their intrateam practices have been all post-lockout (12 practices, as their announcers repeatedly stressed).

The ironic note was, of course, watching the complete role-reversal that I had become accustomed to as a child watching 49er-Saints matchups. The Saints had always been the underrated, expected-to-lose, forever up and coming team for whom the local announcers always had to go out of their way to find bright spots. The 49er sportscasters had the Saints pegged as the feared, dominating monsters that I had always imagined the Niners to be in the Montana, Young, and even the Garcia days. Watching Saints games from my perspective, they have been great these past few years, but history tells me that there are always mountains to climb. The enemy perspective is quite eye-opening in the over-dramatic presentation of the Saints’ strengths.

The two biggest bright spots I took from this game were defensive. First of all, we stopped the run! Mild annoyance and a convenient ability to forget the recent playoff past arise everytime the “greatest rushing play of 2010” is shown – a Marshawn Lynch run up the middle where he broke tackle attempts from all 11 defenders and several sideline players as well en route to a victory-sealing (and Super Bowl repeat destroying) rushing touchdown. However, against one of the better rushing packages in the league, the Saints managed to consistently shut down the run. Comparisons between last year’s squad and the Super Bowl champs of the year before showed consistent inability to stop rushers. So, this is a hopeful, albeit not-to-be-trusted, predictor of 2011 performance. Whether or not Gregg Williams being Gregg Williams was fair in an opening pre-season game, the ability of the defense to disrupt the offense (by whatever means) was impressive. The constant take a chance, strip the ball, harrass the QB tactics that were so lacking from Gregg Williams 2010 model showed up in this year’s model and were a welcome memory of years (2009) gone past.

However, should either of these should be trusted against one of the worst-prepared preseason teams? No, they shouldn’t. Prove yourself, Black and Gold defenders, and I’ll allow you to be put in the same sentence as your predecessors two years removed.

The running game seemed decent, and it was great to see the rushers pounding away into one of the better defenses in the league. Special teams really shone with Joe Morgan securing a spot on at the least some NFL roster this season. There were no 49er returns of any note, and there were only moderate opposing drives that eventually stalled, excepting David Akers succeeding in breaking his career-long field goal distance.

So, what prognostication should we take from this? None. This game will provide adequate fuel to the Niners in future games with the “we were disrespected by Gregg Williams” card. The game will be a temporary ego-boost for the Saints for the next day or two until Sean Payton points out how hollow the win is. Joe Morgan and other potential cuts will eventually make the roster due to their performances Friday. But let’s relax the hype about the game – hyped as it was by being the first game for both teams after the torturous lockout of the past 5 months.

What to look for? The Saints’ preparation in the off-season providing an edge over opponents in the first 3 weeks of the season. The 49ers signing new offensive linemen and really using this game as a motivational springboard. Plenty of wide receivers making the Saints active roster. The continuing defensive swagger that has defined the Saints’ last 2 seasons. Colin Kaepernick’s first start between week 6-12. Early hype about the Saints’ chances in the first half of the season. John Harbaugh revealing his game plan in Week 1.

Facebook outs your phonebook

August 10, 2011

I’ve always been pretty easy on Facebook, letting people share what they want to share and be responsible for the information they post online. People know what they’re posting, and they can be reasonably held accountable for it. I have a few friends – mostly women who’ve had stalkers in their past – change their name on the profile to hide from unwanted attention seekers. Men will apparently do anything to jump at a girl they think is cute, so I don’t blame them. I have a soft spot in my heart for their situation, since they’re pretty powerless against some creep who will do anything in his power to track her down.

Well, my defending of Facebook privacy just ended a half-hour ago. I just found a link to my entire phone book in Facebook and EVERY person who has put their phone number on facebook. Let me repeat that. If you’ve put your phone number in facebook (which is used to verify that your account isn’t fake), and you have EVER given me your number, I can now find your Facebook profile. I have numbers from women I’ve met ONCE and gotten their number, and now I can see their names and any other information on Facebook. Before I say anything else about this, anyone from Facebook reading this, DISABLE THIS IMMEDIATELY. This is completely unacceptable and borders on the worst habits you would expect from cops overstepping their bounds (which doesn’t happen too often – I’m quick to defend officers, as well). To give a step-by-step account of why this is a violation on way too personal a level: If I never gave Facebook my phone number, I’d have to fill in CAPTCHA’s (the box where I have the read the screwed up letters and type them in) a bunch of times when doing common facebook activities. So, I gave them my phone number. Then, I installed Facebook for my smartphone. I didn’t really care at the time, but they make you accept their access to your phone so they can send you background pictures of people who call you from their facebook profile pages.

Well, I’ve been able to go through my list of past phone contacts (even those I’ve deleted), and I’ve seen some names I hadn’t seen before. Of these names, Facebook found about 30-50 people whose number I may have written down once and linked me to their profile, including their full name. For example, I found out that someone I put in my phone as “Jessi [Local Bar Name]” now lives in New York, is friends with a friend of mine, and works for a major retailer. If I was a creepy stalker, that girl could have now had 1) phone calls from me knowing her name, 2) questions from her friend about her or her job, 3) inquiries at her job…. The creepy list goes on. And women face this kind of creepy shit all the time. Facebook, don’t give these people more avenues to harrass people that want to hide their personal lives from the world.

I found a perfect example – I met a girl at the pool in Vegas a few months ago, and we realized we don’t like each other. Now, I know her full name! She never wanted to give me that. By giving me her phone number, she was able to screen my calls and control all the information she allowed me to know. Not anymore! Thank you, Facebook!

More examples, an ex-girlfriend I never want to see again who stopped taking my calls – found her. About 15 numbers (girls and guys) that I never got around to deleting have obviously changed. Unless my old drinking buddy Sean changed his name to Moesha and got a boob job. And I’ve never known anybody named Prettyboy K******.

Let’s take this exercise in invading people’s lives a step further. This means all I have to do is find out a person’s number, put them in my phone, and on the next download, I can find their Facebook profile? Seriously, Zuck?

I’ll be honest, I’ve enjoyed Facebook, because I get to keep up on everybody I can’t otherwise keep up with. Facebook has made high school reunions obsolete, allowed me to see pictures of my family without flying back to New Orleans, let me see my best friend’s house and new kid in Austin, and all kinds of truly wonderful things. But these people volunteered ALL of this information. Please don’t allow me to peek behind others’ veil of privacy without their permission. It’s disrespectful at least and harmful and dangerous at worst.

I’m not mentioning my phone type or the link to this “feature” / invasion of someone’s life, for fear of stalkers tracking down my friends or yours.

Founders Gone FAILED

July 29, 2011

We’ve all heard of or been involved in stories about startup founder conflicts or other things that tear apart startup co-founders. They’re usually pretty fascinating because they reveal psychological differences or poor relationships or different life goals or any number of other facets of interpersonal experiences.

So, during a night of tech discussions at AppHarbor’s new HQ, I was amused by a comment from one of the participants, “In the middle of driving across the country, that’s when the startup died.”

“Wait, did you say the starter died, like the car’s starter?”

“No, the startup died.”

What followed was a smirk-evoking story of a week-long drive between these two new co-founders. Yet, along the drive, they discovered that there’s no way they could be business partners. “A guy can seem so normal online, but when there are real pressure situations, he can’t deal with them.” Midway through the together-all-the-time trip, they realized they’d have to part ways.

The story was pretty funny to me, but I could easily relate. I, too, have had a few failed attempts at getting co-founders. Despite my best efforts, people that are interested in developing the idea either find other commitments or display qualities that would be destructive to a startup. In the story above, it was pretty obvious that if his new partner couldn’t handle a little pressure during a cross-country drive that he would never cut it in the fast-paced, high-pressure startup world.

My first startup attempt was a clone of LiveChat, a site offering webchat as a service. Living in New Orleans in 2004, I met a guy who had this idea and had tried farming the idea out to be developed in India. The developers held his code for ransom, so he was trying to start over with a new partner. I agreed to partner with him and created ClickNChat over the next month. However, I found I was building this product, and he wasn’t living up to his end of the bargain, which was getting customers, marketing, etc. Eventually, I got tired of building a product that wouldn’t get used in the marketplace and gave him an ultimatum. We parted ways soon thereafter.

Years later, I had my idea for FundFill while eating lunch with my best friend at the time, also a .NET developer. We were pretty excited by the idea, and we quickly developed it into a website idea and a business plan. Unfortunately, we lived in a technologically impoverished area, and we worked for someone who had no concept of what the tech world was about. Louisiana, it turned out, wasn’t very fertile ground for building a tech startup. So, the CEO caught wind that we were discussing the idea of a startup (I had talked my friend into forming an LLC with me). LLC’s are public information in LA with everyone’s name on it, and I was fired when news of the LLC was secretly mailed to the CEO. (Ok, stop laughing, I was ACTUALLY FIRED for doing a website in my own spare time!) Unfortunately, my firing had its desired effect – my friend was too scared to do any more work on my project. We continued the flirtation of him joining the project for awhile, but it would always build up my hopes only to dash them again when when he couldn’t contribute to the project.

It was a pretty demoralizing killer to all my momentum every time I tried to recruit him again, so I eventually stopped discussing the project entirely with him. Moving to the Bay Area, I looked around for like-minded people and was delighted to find plenty of developers working in startups or interested in them. In January, a friend who knew about my project introduced me to another guy who had built up his own side-project website. I introduced the new guy to the concept of Fundfill, and he was immediately excited. I was pretty excited myself to have a new business partner to help build my project, now well on its way. Over the next week or two, I introduced him to all the concepts and future plans behind FundFill, and we had even come up with a new feature we could add to the site. We had pretty good chemistry but one big problem. Somehow, my friend who introduced us had become involved in the project as well. He had some qualities that I didn’t want to have in my future company, and I felt stuck, because it seemed it was becoming an all or nothing deal. Furthermore, there was a strong push towards moving the project into a smaller niche than I was comfortable with, given the huge potential of FundFill. I was in a real bind, because I really wanted help on the project, but I didn’t want to give up my business goals. Rather than jeopardize the potential of the project by taking on a partner I was uncomfortable with and shrinking the project’s potential, we parted ways.

So, where am I today? I put up the first fund on FundFill Sunday, and I’m working on new features and expanding my user base. In spite of my setbacks, I used the failed partnerships as new motivation to build my site up. In spite of this post’s negative experiences with finding partners, I’m still keeping an eye out for potential partners to be co-founders (interested in solving technical problems and solving the world’s problems? Check out FundFill and how it works, and e-mail me).

Got a horror story about “Founders Gone FAIL”? Leave it in the comments. (I know you’ve got some pretty good stories!)

What is FundFill?

July 8, 2011

One of the challenges of my side startup project, FundFill, has been explaining what it is in 30 seconds or less. Yet tonight, I was able to finally find quick explanations that explained the concept behind it as well as the potential for its use.

FundFill is a demand-based marketplace, a bounty-driven marketplace, where you can create a fund, for as little as a dollar, towards anything you want. If other people are interested in helping fund this idea, goal or accomplishment, they can add money into the fund, raising its pool of money. Once the fund is accomplished (according to the goal you describe), the person who accomplishes it will win the money in the pool. For example, if you’d like to see a video of a chihuahua jumping rope, you could create a fund called “Chihuahua Jumping Rope” for $1. If a few more people put a few bucks towards the idea, the fund could now be worth $100 to the first person who posts this video fulfilling the fund.

The Ansari X Prize was a $10 million prize for the first corporation to put a rocketship into space twice. Funded by corporations, this spurred demand for companies across the globe to try to accomplish this goal. Why should only corporations be able to create these prizes? FundFill is a democratic version that allows users to create their own prize or fund, so that the people can help spur what they want to see, not just corporations with big money.

These funds can also be bounties to reward people for helping with something. If a criminal act inspires you to want to help with a crime, you could pledge to a fund for catching the criminal. Interested in a how-to video on snowboarding? Create a fund for that! Any goal – however big or small – can have a fund for it. Want to offer a buck to the first people to colonize Mars? Sure, pledge away! ūüôā

As more and more funds are added to the site, users can browse through existing funds to find something they’d like to help fund. The more popular an idea is with the users, the more money that fund will gather. And the more money the fund has, the more likely the fund is to be completed. This creates a unique demand-based marketplace that doesn’t really exist elsewhere in the real world.

Want to make an idea more likely to be fulfilled? Add more money to it. See enough money in a fund for you to accomplish it? Perform the task described in the fund and apply to win the money.

Create a fund, fulfill a fund, win money.

Fund it! Fulfill it! Win it!

Launching soon! fundfill.com

C# equality

June 3, 2011

Look at the following three and see if you can figure out which one works correctly for finding any single common item in a set:

1) Permissions.Intersect(permissions).Any();
2) Permissions.Any(p => permissions.Contains(p));
3) (from p in Permissions from p1 in permissions where p1.Equals(p) select p).Any();

Note that I’m trying to do comparison on the Equals method, not by reference.

If you guessed that I posted these in the order I tried, you win. The third try was the one that finally allowed me to compare two groups to find if there were any common items. I’m kind of annoyed that I had to go through these three iterations. It’s always a crapshoot whether methods will compare by reference, equality or the Equals() method.


Alt.Net San Francisco – dinner

May 14, 2011

I attended Seattle’s Alt.Net conference last weekend, and I was quite impressed with the community up there. For all the awesome geekiness they were able to bring to the table, I was amazed and perplexed that San Francisco doesn’t have the same thing. After searching, tweeting, and making phone calls looking for information on this elusive community, I came to the conclusion that it doesn’t exist. Alt.Net, the major alternative community to the Microsoft hegemony over the .Net world, is nowhere to be found, or possibly just defunct, in the technology center of the planet! (I’m hoping I’m wrong, and I’ll have a comment below that it does exist).

I’m jealous, and I’d like to be able to have the community, conferences, knowledge sharing, expertise, and networking provided by a strong Alt.Net community. One of the main advantages of Ruby, OSS, Python, and other technologies here is their strong communities. There’s no reason we can’t do the same thing!

So, in order to get this off the ground, I want to host a dinner somewhere in San Francisco for Alt.Netters to come together, discuss the community, and interact with others like us. I’m proposing a Monday night dinner sometime in the next month (possibly May 23rd, 2011). Hopefully, we can discuss everyones’ experiences with the .Net community in the Bay Area, and figure out what people would like out of it. Ultimately, I’d like to be able to build up a community as strong as Seattle’s so that we could host our own Alt.Net conference here in the Bay Area.


1) Well, for now, I’d like to get your participation in the initial dinner. Contact me on Twitter (@joebalfantz), by e-mail (jbalfantz upon gmail), or comment below. If May 23rd is horribly bad for most people, I’ll create a different date. I’ve created a meetup.com group for this: http://www.meetup.com/Alt-Net-San-Francisco/.

2) What do you want out of the community? I’ve searched around for groups for 3 years in the Bay Area, and I’ve found a few decent ones, but I really found what I was looking for in that Seattle conference. Well, I’m not moving there, so I want to re-create that in The City. What do YOU want out of it?

A new love

February 8, 2011

A very few times in my life I’ve been in love, and I’ve noticed there’s a certain sigh that accompanies this feeling. Walking to work, I sighted her again, and I caught myself mid-sigh and wondered, “did that really just happen?” Yup, I’m guilty as charged. I love the city of San Francisco. In the last week, I’ve done alot of walking and running around her and caught the beautiful sights from new angles, but I can’t say I limit my liking to pure external beauty – that sort of relationship, I’ve found, can only get one so far. It also extends to the people and the attitude and the vision captured here.

I’ve found the largest concentration of people who give a shit and who dream and care about not just their narrowly-focused goals and ambitions but about the greater consequences. This mindset allows an unselfish perspective of the world as a whole and not just “how is the going to affect me?” I’m a dreamer, I see the bigger picture and dream bigger, more impossible dreams because I know in my heart I can accomplish them. Here, I’m surrounded by like-minded people who believe they can make a difference in the world. We believe that the power of a few can build things and shape things that can affect millions and billions of lives. You only live once, so it’s no use living in the small scope. In experiments, if you drop a creature in an environment where it’s growth is hampered, his main concern is simply to survive and keep his life prolonged with what little resources he has. Drop the same creature in an environment rich with the resources he needs and he will flourish and thrive and be free to worry about more than survive. This creature is my dreams, my ability to create and foresee better things in all we do. And this city is the wonderful environment that allows me to stop arguing whether you can changes things and start debating how to change them and how quickly. The resistance against my dreams is gone here. Ah, amore!

I find it fascinating how this city isn’t a realistic city; it’s more of a fairytale, made-up land that shouldn’t exist. By circumstance and geography, San Francisco has a large population of very intelligent and wealthy individuals, allowing the government to be this crazy progressive, unrealistic, pie-in-the-sky crew that get to experiment excessively and try things that the rest of the country points their fingers at and says “really? Crazy liberal San Francisco!” Personally, I find myself liking some of the ideas and really hating some of them (how did we not pass prop B and close a future budget nightmare! Aaaargh!!!). Still, I accept I live in a fairytale land that no other city could mimic, due to the money they’d never receive.

I’m an inventor, someone who dreams and imagines and actually create new realities, and I’m surrounded by my mind kin. This is a match made in heaven.

Clipper Clips me

August 10, 2010

Interesting situation I’ve gotten myself in now that San Francisco MUNI no longer sells monthly passes directly. Beginning in August, they’ve changed to force everyone who wants a monthly pass to use the Clipper card (formerly TransitLink). Now, I have a monthly reminder to purchase a new MUNI pass on the 12th of the month (the second day it’s available). So, I purchased my Clipper card for the first time on the 14th. Now, I may have missed reading the receipt hard enough, but MUNI and Clipper rules say that any monthly pass bought before the 16th is for the current month and afterwards is for the next month. So, I didn’t realize I bought a pass for the month I was already in. Oops. Since your MUNI grace period for the following month ends after the third, I got an insufficient funds notice on the 5th. I called Clipper to discover that my pass I purchased had been for July. Great, so how do I get that pass changed to August, I asked Clipper customer service. Your usage records should show I never used the pass in July to prove I’m not trying to create a scam. However, I’m told it’s tough shit, and that I could purchase another pass for August. Really? I’m not buying another pass when I haven’t been able to use the one I’ve got. The guy laughs when I ask for his manager (who I’m told won’t get in until 9AM). Calling MUNI customer service, I got more of a surprised tone that Clipper didn’t handle this, and I was able to lodge a complaint with them, just to make them aware that this is the first month of Clipper, and maybe Clipper should be handling this better.

So, I waited until after 9AM on Monday and called in, hoping to get this resolved before I had to bus to work. This guy helped me right away and said he would have to get approval for a refund to my account. However, the refund approval had to go through MUNI, which may take 3 weeks. If today is the 9th, and three weeks from now is the 30th, doesn’t that kind of make this pointless. The guy says the refund needs to get approved by MUNI, since Clipper can’t take the rejection of the refund, meaning they would have to fork out $60. I appreciated that this guy actually tried to help, and I had him begin the approval process, but come on, this is a ridiculous way to handle someone being proactive enough to purchase a pass early.

Feeling stuck but searching for a compromise that would allow me to ride the bus without carrying around stripper wages, I offerred to buy another pass for August, and when MUNI processed the refund, I’d have that credit applied to my next month. This solved the problem, but I’m a little disturbed by the process and not having a better failsafe in place for the first month of Clipper. A simple approval process should be in place to help out people who unwittingly purchase the wrong month. MUNI should specify some guidelines for Clipper to follow such as a) verify that the card is valid, b) if the card was not used in the month the customer complains is the incorrect month, the pass should be transferrable. The simple advice: whenever a new system is implemented, always expect problems. They will ALWAYS show up.

Problem with Event Sourcing – reconstituting our objects

March 12, 2010
This is a question in response to Greg Young’s blog post CQRS and Event Sourcing:

It seems that a pretty big difference between Event Sourcing versus traditional DDD using an RDBMS is that one focuses on persisting the events, rather than snapshots of the objects. ¬†A major advantage here is that you have a more enriching data experience – saving the events allows you to not only know the objects and states, but also all the events that took place to get you to your current place. ¬†Imagining your domain’s current snapshot of data as a two-dimensional object, event sourcing gives you a third dimension of perspective. ¬†It doesn’t give you a static, current snapshot, but can provide the actions and history – giving you a much richer experience of being able to use the data. ¬†However, I have an issue. ¬†Sometimes, the snapshot is exactly what I want. ¬†I may need to know exactly the data from one object and use that in another process. ¬†In a cold state where the given object is not in memory, I want to be able to pull the object and perform some action on that object, which is an interaction with the object and its current state. ¬†So, let’s say I have an object foo that I need to perform some action (this event relies on the object’s current state). ¬†How do I reconstitute that object into its current state efficiently? ¬†If this object has a long history, I may have to reconstitute the object by iterating over all of its events and rebuild the object from scratch. ¬†Or even worse, if the object’s history intermingles¬†with other objects and their events, I have to pull multiple objects’ histories and perform all the same actions against them to get the object caught up to its current state. ¬†In an event-only storage system, this third-dimesional domain perspective now works against me. ¬†In a dual-storage system, at least, you still have the latest copy (albeit your argument is that these current-state-snapshots are not as reliable as the events).

Is there a trick to pulling the most current snapshot of an object into memory? ¬†Or do you discriminate between processes and objects that work better with Event Sourcing and those that work better with object-snapshot storage? ¬†Furthermore, what happens when reconstituting an object based on past events is not a deterministic action (say, the action/event changed or a bug was found in an event that was fixed). ¬†In other words, code is fluid and changes often. ¬†How reliable is a snapshot of an event long ago in an object’s history when the function or process may have changed, however slightly. ¬†There’s no longer a deterministic process that will always generate the same values for our object that will be historically accurate. ¬†Does a change in an event’s actions become an event itself? ¬†I.e., are we now forcing our developers to write “Event_Changed” events? ¬†I started to write a concrete example, but it was complete crap, so if I’m unclear just let me know.

It seemed easier to post a blog than drop this entire thing in the comments section ¬†ūüėõ