Forms are the center of many technical and organizational problems
March 20, 2020
We've seen, across countless organizations, a similar pattern of the life of a product form. Regardless of where you are along the lifecycle, it's useful to be mindful of it.
This should be quick...
So you need to collect some information from your users, and are setting out to build a product form. In all likelihood, it's something that's a key part of your user experience: perhaps even the first conversation you have with a user, in the form of a sign-up flow.
Forms seem like they'd be easy to build. After all, html has supported the
<form> element for ages, and you have access to some developers, so why not let them build? Particularly since your form feels pretty unique, given your organization's unique product or pitch, and the branding aesthetic you have in mind to support it, it seems that nothing off the shelf would work for this.
You get together with your designer, a couple of stakeholders, and in a couple of hours come together with a user flow - a handful of post-it notes on a wall, some presentation slides, and some copy to go alongside a design that's likely inspired by work that others have done.
You're feeling confident, this should be a quick project and then you'll be able to move on.
It's not perfect, but it's good enough.
Since this probably isn't the only thing your company has going on, you might end up needing to wait a few days or weeks for developers to get involved. But once you're at the top of the list, you hand off the project to the developers, you provide them the content, design, as well as some wise guidance like "we'll want to test changes to this in the future" and "make sure that it's responsive", and "we might want to make parts of this dynamic".
The developers are initially excited about the project. Unlike working on backend work, or cobbling together marketing pages, this feels like something that will directly speak to users. Plus, most everyone on the team has done some variation of forms in the past, and it seems to them that they'll be able to string together a few off-the-shelf libraries to put the flow together, and they'll be done. Wise engineers will also take a little time to architect a small framework for managing form state, and it's off to building.
However, as the project is underway, they start encountering hiccups. The many existing form frameworks and validation libraries don't play well together. There's a question of what should be done on the frontend, and what should be done on the backend. You're not really sure how the flow will evolve in the future, so you eschew making complicated schemas in favor of down-the-line flexibility. Everyone starts to get a feeling of re-inventing the wheel, poorly.
As many complicated technical issues of state management, data integrity, and logic get encountered, the short initial estimates for the project timeline result in those issues being punted down the line, and some "nice to haves" get cut out as being "out of scope". Business counterparts get told: we'll be able to fix things and change them later, since we designed the system well.
A bit delayed, but ready to go.
After a few weeks, it's good enough, you slap in google analytics to understand what's going on, and you're ready to go live!
Everyone's excited! Your onboarding flow is live. You catch a couple of small issues but you're pretty happy with it overall, so you take a little bit of time to celebrate.
Ah… missed a couple more things, just some low-hanging fruit.
Once you're live for a few days, little issues start popping up. Not quite unexpected, but there seem to be a lot of them. Some executives complain it doesn't work on their home computers. Developers forgot to make the back button work. The inputs seem to jump around in some cases. It's hard to read some of the pages on smaller screens.
At first it seems like a lot of low-hanging fruit. But there's a lot of this low hanging fruit. You fix a few of the first problems, but then it gets hard to address the bigger issues, since the basics work after all, and inevitably the form project gets deprioritization since the company has other things to work on. After all, you're trying to run a business, not a form company.
Seems like we'll need to address some of that technical debt.
After a couple weeks, you start running into some bigger problems with the forms. You realize that it's pretty difficult to actually analyze the flow, and you'll need to redo the marketing events. Some customers start the flow, go away, and then come back later, and their experience is broken. And then a junior developer makes a small change to the flow logic, and accidentally breaks the ability to complete the signup. Minor changes seem to take on the complexity of performing surgeries, if surgeries were performed in the dark.
There's a mounting need to address some of the technical debt, but there's no time to. After all, the project is done, and you've moved on.
Turns out we didn't really understand the user.
Once the data starts coming in, and you hear from real users, it turns out, the meeting a few months ago where we sketched out the user journey wasn't entirely correct. Perhaps your product isn't suitable for a single-shot purchase, and people are re-visiting your site multiple times before converting. Or you've realized that your idealized persona of a "user" is actually dozens of different user types -- some, very experienced, technical users who just want to breeze through, while others need a bit more handholding. People care about different things, but you're putting them all in through the same flow.
People want quicker iterations. Can't we can we just test things? But the engineering time isn't focused on the onboarding flow -- changes are risky, and take a bunch of time as tests must be rewritten, logic tested manually.
Yikes, this form is a mess, as is the team that built it.
While all this is happening, the world around doesn't stop. Your organization's design and brand evolves over time. Marketing wants to try new experiences with new customer segments. Browsers, devices, and the web platform evolve in parallel. Data people want to consume more data in different ways. People start setting up landing page tools, experiment frameworks, and analytics providers, hoping that something will be a magic bullet to make the forms work well.
Organizational discontent starts to brew. Engineers don't want to make small changes to the flow, because they're risky, and they'd rather build more interesting things. But they don't have the time to do the big project of building a full out forms CMS. Executives wonder why they can't have the latest shiny thing. Marketing folks lament how difficult it is to do personalization, or just improve conversion in the first place. Product people are trying to make everyone happy, but can't.
Okay, we need to build it again, hopefully we'll get it right this time.
Eventually, the frustration boils over.
You end up re-building your whole onboarding. Your initial assumptions weren't correct, but this time you'll get it right. You have another series of meetings, you re-prioritize the form project, and you end up spending a couple of months building a new thing, while keeping the old thing running, making sure to also transition smoothly from the old to the new, now that dozens of poorly-documented downstream integrations and dependencies exist.
This takes way longer than the first time. And while you've addressed some of the problems with the v1, the v2 comes with its own problems. You still don't have the institutional buy-in to build a proper forms platform (after all, you're not a forms company) so you end up with another cobbled-together second system.
Guess this stuff is harder than it seemed...
By this point, you realize that this is going to be an ongoing process. Now, you've staffed a whole team of top engineers, product managers, and data people to work on your form flow. That quick project you thought would be done in a few weeks is now a line item in your staffing budget. It's costing you hundreds of thousands of dollars, yet it feels like it's moving slower than ever before.